Software and metadata structures for distributed and interactive database architecture for parallel and asynchronous data processing of complex data and for real-time query processing

ABSTRACT

The various embodiments of the invention provide software and metadata structures for a data processing system and method, for applications such as marketing campaign management, speech recognition and signal processing. An exemplary system embodiment includes a first data repository adapted to store a plurality of entity and attribute data; a second data repository adapted to store a plurality of entity linkage data; a metadata data repository adapted to store a plurality of metadata modules, with a first metadata module having a plurality of selectable parameters, received through a control interface, and having a plurality of metadata linkages to a first subset of metadata modules; and a multidimensional data structure. The control interface may modify the plurality of selectable parameters in response to received control information. A plurality of processing nodes are adapted to use the plurality of selectable parameters to assemble a first plurality of data from the first and second data repositories and from input data, to reduce the first plurality of data to form a second plurality of data, and to aggregate and dimension the second plurality of data for storage in the multidimensional data structure.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation-in-part of and claims priority toNevzat Hurkan Balkir et al. U.S. patent application Ser. No. 11/137,618,filed May 25, 2005, entitled “Distributed And Interactive DatabaseArchitecture For Parallel And Asynchronous Data Processing Of ComplexData And Real-Time Query Processing,”, which is commonly assignedherewith, the contents of which are incorporated by reference herein,and with priority claimed for all commonly disclosed subject matter (the“related application”).

FIELD OF THE INVENTION

The present invention, in general, relates to database managementsystems, and more particularly, relates to a software and metadatastructures for distributed and interactive database architecture forparallel and asynchronous data processing of complex data and real-timequery processing, with a plurality of applications such as, for example,marketing campaign management, scientific research, and speech andsignal processing.

BACKGROUND OF THE INVENTION

Data processing of complex types of data, such as processing of millionsto hundreds of millions of records of individuals, businesses and otherentities, has historically been performed as a batch process using largemainframe computers. These large quantities of data were typically inputinto the processing system using a physical medium such as magnetic tapeor electronic/magnetic disk. Once data processing began, the processwould continue largely uninterrupted, over the course of several hoursto several weeks, and output data would be provided.

For marketing campaigns such as catalogue mailings, promotional mailingsand offers, a client or other system user would request the use ofvarious databases and mailing lists, as input data, and would then berequired to specify, in advance of data processing, how the data shouldbe segmented to provide the resulting campaign list, such as a mailinglist. For example, the various lists and databases may include hundredsof millions of records of individuals, while the resulting campaignwould be for a mailing to 10,000 individuals who meet certain criteria,such as home ownership, previous purchasing patterns, and so on.

Similarly, in various scientific and medical research areas, such asphase three drug evaluations, huge amounts of data may be generatedwhich must be processed to detect various statistical patterns, such asefficacy in a larger population, dosage requirements, significant sideeffects and interactions with other drugs. In addition, many studies areconducted in numerous locations, with data collected throughout theworld. Again, vast quantities of data must be processed, and must resultin a selection of individuals who meet certain criteria, such as havingcertain adverse reactions.

In other areas such as speech and signal processing, vast quantities ofdata may be collected and must be analyzed. For accurate speechrecognition and speech generation, vast data stores may be generated,for thousands of analog electronic signals which must be digitized andparsed into corresponding phonemes, for thousands of words, forthousands of sentences, in any of numerous languages, each withpotentially different pitch, timing and loudness (collectively,prosody), each with different co-articulations based on preceding andsubsequent words and phonemes, and each from thousands of individuals.In addition, huge amounts of data to be analyzed may be collected, suchas for intelligence services to analyze speech signals received frommobile communications for potentially unlawful or dangerous activities.Again, vast quantities of data must be processed, and must result in aselection of words and corresponding pronunciations that meet certaincriteria, such as having a likelihood of fit to selected phonemepatterns from a plurality of different speakers of a plurality ofdifferent languages, with high discrimination and noise immunity.

Because of the batch processing environment of the computing systemsrequired to manage such large data volumes, in the prior art all suchsegmentation or other selection criteria had to be specified in advance.Unfortunately, the selection criteria may not be known in advance,particularly where the determination of the selection criteria is itselfdependent upon the accumulated data, such as in areas of marketingcampaigns, scientific research and speech and signal processing. Inaddition, based upon the data results, a user may want to modify theselection criteria, and is unable to do so in prior art systems, withoutrepeating all of the processing with the modified criteria.

Other prior art forms of real-time data analysis have largely beenconfined to significantly less complex data types, typically solelynumerical data, such as sales and revenue data, capable ofstraightforward arithmetic and algebraic manipulations (e.g., sums andaverages) and numerical methods of analysis (e.g., Riemann summation).Prior art data analysis systems have not succeeded at providingreal-time analysis of more complicated data, particularly complex datawhich requires set operations and not arithmetic manipulations. Forexample, prior art data analysis systems have not allowed for real-timedata analysis of voluminous personal attribute data for marketingcampaign determination and management, to provide a resulting set ofindividuals or households who meet certain criteria, particularly wherethe criteria may be determined dynamically and interactively, inreal-time.

A need remains, therefore, for a database system architecture which canprocess such vast amounts of complex data, in parallel andasynchronously for higher data throughput, which provides for setoperations, and which allows real-time query processing for userinteractivity, such as for data analysis and modifying selectioncriteria. Such a database system architecture should be capable ofprocessing complicated data types, from personal attribute data tospeech and signal processing data.

SUMMARY OF THE INVENTION

The various exemplary embodiments of the present invention provide adata processing system and architecture for distributed processes whichcan be performed in parallel and asynchronously, not only providing forhigh speed data processing, but also providing effective and valuabledecoupling of the provision of output data from the provision of inputdata. The various exemplary embodiments utilize a new form of metadata,which is not only utilized to define data relationships and datastructures, but also is utilized to define processes to be performed onthe data. The inventive metadata of the various exemplary embodiments iscomprised of a plurality of parameterized metadata modules and metadatalinkages, providing for both dynamic and static parameters, and allowingthe various systems and processes to be fully customizable, fullystandardized, and everything in between.

The exemplary systems and processes provide for user interactivity, witha user enabled to provide input, and make selections and decisions,throughout extensive data processing, and to modify previous decisions.Furthermore, such selections and decisions may be made of the basis ofactual data analysis and segmentation, provided through real-time userqueries and system query responses. The exemplary systems and processesallow for the introduction of artificial intelligence processing, withan analytic recommendation processor or engine (also referred to as an“information consultant”) provided to aid in the various user selectionsand decisions, also based on actual data analysis and segmentation.

The various exemplary embodiments may be applied to a wide variety ofdata and signal processing applications, including marketing campaignmanagement, speech and signal processing, and scientific research.

In a first exemplary embodiment, a data processing system for marketingcampaign management, comprises a plurality of data repositories, acontrol interface, a multidimensional data structure, and a plurality ofprocessing nodes. A first data repository of the plurality of datarepositories is adapted to store a plurality of entity and attributedata. A second data repository of the plurality of data repositories isadapted to store a plurality of entity linkage data. A metadata datarepository of the plurality of data repositories is adapted to store aplurality of metadata modules, in which a first metadata module of theplurality of metadata modules has a plurality of selectable parametersreceived through the control interface. The plurality of processingnodes are adapted to use the plurality of selectable parameters toassemble a first plurality of data from the first and second datarepositories and from input data, to reduce the first plurality of datato form a second plurality of data, and to aggregate and dimension thesecond plurality of data for storage in the multidimensional datastructure.

In the various exemplary embodiments, the control interface is furtheradapted to modify the plurality of selectable parameters in response toreceived control information. The various exemplary embodiments may alsoinclude a data input adapted to receive information from a plurality ofdata sources selectable through the control interface. The controlinterface may further comprise a user interface which is adapted toprovide for selection of the plurality of selectable parameters,selection of a plurality of attributes for data processing, selectionand ordering of a subset of data processes of the plurality of dataprocesses, and selection of a plurality of dimensions for aggregation.

The first metadata module has a corresponding plurality of metadatalinkages to a first subset of metadata modules of the plurality ofmetadata modules, wherein the corresponding plurality of metadatalinkages comprise at least one of the following metadata linkage types:a communicated parameter of the plurality of modifiable parameters, acommunicated attribute of a plurality of attributes, a processdefinition, a process flow, or a process communication.

In the various exemplary embodiments, the plurality of processing nodesmay be adapted to perform in parallel and asynchronously at least two ofthe following processes: a data input process, a data layout process, adata conversion process, a data correction process, a data auditprocess, a data matching process, a linkage assignment process, a datacoding process, a data enhancement process, a data change detectionprocess, or a data update process. The plurality of processing nodes mayalso be adapted to use modeled information to provide a version of theplurality of selectable parameters. In the various exemplaryembodiments, the plurality of processing nodes may be further adapted toreduce the first plurality of data to form the second plurality of data,in parallel and asynchronously, and to store in an associated memoryinterim results comprising entity linkage data having a plurality ofpersistent identifiers, using at least one of the following processes: adata netdown process, a data suppression process, or a dataprioritization process.

In the various exemplary embodiments, at least one processing node ofthe plurality of processing nodes may be adapted to process a query andprovide a query response, using the multidimensional data structure. Inaddition, in exemplary embodiments, at least one processing node of theplurality of processing nodes may be adapted to determine a plurality ofsets from the multidimensional data structure and to perform a pluralityof set operations on the plurality of sets, the plurality of setoperations comprising union operations and difference operations, suchas to determine a plurality of mutually exclusive sets from a pluralityof sequential union sets by forming a difference set by subtracting aprevious union set from a next union set.

In the various exemplary embodiments, the plurality of processing nodesmay be further adapted to use a modified plurality of selectableparameters to reduce the first plurality of data to form a modifiedsecond plurality of data; to use the modified plurality of selectableparameters to aggregate and dimension the modified second plurality ofdata; or to use a modified plurality of selectable parameters tore-aggregate and re-dimension the second plurality of data.

Also in the various exemplary embodiments, at least one processing nodeof the plurality of processing nodes may be adapted to provide aplurality of dimensions for aggregation of the second plurality of data,or adapted to dimension the second plurality of data using a pluralityof attribute ranges selectable through the control interface.

Also in the various exemplary embodiments, the plurality of processingnodes are further adapted to provide a marketing campaign fulfillmentprocess, in which the marketing campaign fulfillment process comprisingat least one of the following processes: report generation; postalservices; postal presort services; inkjet services; printing preparationservices; shipping services; data extraction; email formatting;generation of a magnetic tape, disk or electronic file having selectednames and physical addresses; generation of email addresses; selectedcontent determination; selected communication channel determination,variable formatting, and variable layout.

In the various exemplary embodiments, the system may also include a dataprocessing selection and planning node coupled to the plurality ofprocessing nodes, the data processing selection and planning nodeadapted to provide the plurality of selectable parameters in response touser input.

Also in the various exemplary embodiments, the plurality of metadatamodules define the plurality of data processes and further define aplurality of data structures stored in the plurality of datarepositories. The plurality of metadata modules may be comprised of atleast two metadata modules of the following group: a parties module, anaccounting module, a hardware resource module, an activities definitionmodule, a service invocation module, a process definition module, amarketing campaign definition module, a list selection module, a listlayout module, and a data enhancement module. In selected embodiments,the marketing campaign definition module has a metadata linkage to theactivities definition module, wherein the activities definition moduledefines a plurality of activities comprising a selected marketingcampaign; the activities definition module has a metadata linkage to theprocess definition module, wherein the process definition module definesthe plurality of data processes comprising a selected activity; and theparties module and the activities definition module each have a metadatalinkage to the accounting module, wherein the accounting module trackscosts and expenses associated with a plurality of activities for aselected party of a plurality of parties defined in the parties module.

The entity linkage data may comprise plurality of persistent identifierslinking individual or household entities to a corresponding plurality ofattributes, the corresponding plurality of attributes comprising atleast two of the attributes of the following group of attributes: name,current address, name and address variations, previous address, age,gender, income, social security number, vehicle ownership, maritalstatus, spouse name, spouse address, children names, children addresses,household identification, home ownership, purchasing histories,marketing responses, and employment histories.

Also in the various exemplary embodiments, the plurality of datarepositories comprises one or more databases having relational schema, adata warehouse having relational schema, and a data mart having a staror snowflake schema; and the multidimensional data structure has amultidimensional, on-line analytical processing (OLAP) data cube schema.The plurality of data repositories may be embodied as one or more of thefollowing group of data storage devices: a stored area network (SAN)magnetic disk array, a memory, a mainframe file system, partitioneddatabase storage, a Unix file system, or a Windows file system; whilethe plurality of processing nodes may be embodied as one or more of thefollowing group of processors: a mainframe processing node, a clustereddatabase node, a Unix application node, or a Windows application node.

In another exemplary embodiment, a data processing system includes: acontrol interface adapted to receive a first plurality of selectableparameters, a second plurality of selectable parameters, and a thirdplurality of selectable parameters, the control interface furtheradapted to modify the first, second and third pluralities of selectableparameters in response to received control information; a data inputadapted to receive input data; a data and messaging network coupled tothe control interface and to the data input interface; a first datarepository coupled to the data and messaging network, the first datarepository adapted to store a plurality of entity data and a pluralityof corresponding entity attribute data; a linkage data repositorycoupled to the data and messaging network, the linkage data repositoryadapted to store a plurality of unique and persistent identifierswherein each persistent identifier corresponds to each unique entity; adata assembly processor coupled to the data and messaging network, thedata assembly processor adapted to use the first plurality of selectableparameters to generate a first plurality of data from the first datarepository, from the linkage data repository, and from input data; asecond data repository coupled to the data and messaging network, thesecond data repository adapted to store the first plurality of data; adata reduction processor coupled to the data and messaging network, thedata reduction processor adapted to use the second plurality ofselectable parameters to generate a second plurality of data from thefirst plurality of data; a third data repository coupled to the data andmessaging network, the third data repository adapted to store the secondplurality of data; an aggregation processor coupled to the data andmessaging network, the aggregation processor adapted to use the thirdplurality of selectable parameters to aggregate and dimension the secondplurality of data; and a multidimensional data structure coupled to thedata and messaging network, the multidimensional data structure adaptedto store the aggregated and dimensioned second plurality of data.

In another exemplary embodiment, a parallel and asynchronous dataprocessing system for marketing campaign management includes a userinterface; a control interface; a plurality of data processing nodescoupled through a data and messaging network to the user interface andto the control interface, the plurality of data processing nodes adaptedto process data; a first data repository coupled through the data andmessaging network to the plurality of data processing nodes, the firstdata repository adapted to store a plurality of entity name and addressinformation, and a plurality of corresponding entity attributeinformation; a linkage data repository coupled through the data andmessaging network to the plurality of data processing nodes, the linkagedata repository adapted to store a plurality of unique and persistentidentifiers wherein each persistent identifier corresponds to eachunique entity; a second data repository coupled through the data andmessaging network to the plurality of data processing nodes, the seconddata repository adapted to store a first subset of information from thefirst data repository and the linkage data repository, the first subsetof information including a first subset of entity attribute information;an aggregated and multidimensional data structure coupled through thedata and messaging network to the plurality of data processing nodes,the aggregated and multidimensional data structure having an aggregationof the first subset of information dimensioned with a first plurality ofselected attributes of the first subset of entity attribute information,wherein the first plurality of selected attributes are adapted to bemodifiable during data processing through the user interface or thecontrol interface; and wherein the plurality of data processing nodesare further adapted to determine the first subset of information storedin the second data repository and to aggregate and dimension the firstsubset of information using the first plurality of selected attributes.

In another exemplary embodiment, a data processing method for marketingcampaign management includes: storing a plurality of entity andattribute data in a first data repository of a plurality of datarepositories; storing a plurality of entity linkage data in a seconddata repository of the plurality of data repositories; receiving aplurality of selectable parameters; storing a plurality of metadatamodules in a metadata data repository of the plurality of datarepositories, a first metadata module of the plurality of metadatamodules referencing the plurality of selectable parameters; using theplurality of selectable parameters, assembling a first plurality of datafrom the first and second data repositories and from input data; usingthe plurality of selectable parameters, reducing the first plurality ofdata to form a second plurality of data; using the plurality ofselectable parameters, aggregating and dimensioning the second pluralityof data; and storing the aggregated and dimensioned second plurality ofdata in a multidimensional data structure.

In yet another exemplary embodiment, the present invention provides atangible medium storing computer readable software for programming aparallel and asynchronous database architecture and data processingsystem for execution of marketing campaign management and analysis,comprising a plurality of program means, such as computer- orserver-executable instructions. The plurality of program means includefirst program means for receiving a plurality of selectable parameters,for modifying the plurality of selectable parameters in response toreceived control information or in response to modeled information toform a modified plurality of selectable parameters; second program meansfor storing a plurality of entity and attribute data in a first datarepository of a plurality of data repositories and for storing aplurality of entity linkage data in a second data repository of theplurality of data repositories; and for storing a plurality of metadatamodules in a metadata data repository of the plurality of datarepositories, a first metadata module of the plurality of metadatamodules referencing the plurality of selectable parameters; thirdprogram means for using the plurality of selectable parameters toassemble in parallel and asynchronously a first plurality of data fromthe first and second data repositories and from input data; to reducethe first plurality of data to form a second plurality of data; and todimension and aggregate the second plurality of data; fourth programmeans for storing the dimensioned and aggregated second plurality ofdata in a multidimensional data structure; fifth program means forprocessing a query and providing a query response using themultidimensional data structure; and sixth program means for using themodified plurality of selectable parameters to reduce the firstplurality of data to form a modified second plurality of data; and usingthe modified plurality of selectable parameters to dimension andaggregate the modified second plurality of data.

In yet another exemplary embodiment, the present invention provides atangible medium storing computer readable program instructions forexecution by a computer system for marketing campaign management andanalysis. The tangible medium storing computer readable programinstructions comprises: a first metadata construct of a plurality ofmetadata constructs, the first metadata construct defining a pluralityof data processes for marketing campaign management; a second metadataconstruct of the plurality of metadata constructs, the second metadataconstruct having a corresponding metadata linkage to the first metadataconstruct, of a plurality of metadata linkages, the second metadataconstruct defining a first plurality of relational data structures for afirst plurality of data repositories of the computer system, the secondmetadata construct further defining a multidimensional data structurefor a second data repository for marketing campaign analysis; and athird metadata construct of the plurality of metadata constructs, thethird metadata construct having a corresponding metadata linkage to thefirst metadata construct, of the plurality of metadata linkages, thethird metadata construct defining a plurality of input data sources anda plurality of selectable attributes.

The exemplary embodiment may also include a fourth metadata constructadapted to provide a plurality of modifiable parameters to the firstmetadata construct, the plurality of modifiable parameters providing forselection of the plurality of input data sources and selection of afirst subset of the plurality of data processes for execution by thecomputer system. The plurality of selectable input data sources comprisea plurality of databases and marketing lists containing a plurality ofattributes, the plurality of attributes comprising at least twoattribute types of the following group of attribute types: a nameattribute, an address attribute, a demographic attribute, a behavioralattribute, or an attitudinal attribute.

The exemplary embodiment may also include a fifth metadata constructadapted to execute the first subset of the plurality of data processesin parallel and asynchronously, using the plurality of modifiableparameters and a corresponding plurality of metadata linkages to othermetadata constructs. The first subset of the plurality of data processescomprises at least two data assembly processes of the following group ofdata assembly processes: a data input process, a data layout process, adata conversion process, a data correction process, a data auditprocess, a data matching process, a linkage assignment process, alinkage selection process, a data coding process, a data enhancementprocess, a data change detection process, or a data update process. Asecond subset of the plurality of data processes comprises at least onedata reduction process of the following group of data reductionprocesses: a data netdown process, a data suppression process, or a dataprioritization process.

The exemplary embodiment may also include a third subset of theplurality of data processes which are are adapted to use themultidimensional data structure to process a received query and toprovide a query response.

The plurality of metadata linkages typically comprise at least onemetadata linkage type of the following group of metadata linkage types:a common or communicated parameter of the plurality of modifiableparameters, a common or communicated attribute of a plurality ofattributes, a common process definition, a common process flow, or acommunication between processes.

The exemplary embodiment may also include a sixth metadata construct ofplurality of metadata constructs, the sixth metadata construct adaptedto track each data process invocation of a plurality of data processinvocations; a seventh metadata construct of the plurality of metadataconstructs, the seventh metadata construct adapted to associate a costfactor with each data process invocation of a plurality of data processinvocations; an eighth metadata construct of the plurality of metadataconstructs, the eighth metadata construct adapted to associate aresource usage factor with each data process invocation of a pluralityof data process invocations; and a ninth metadata construct of theplurality of metadata constructs, the ninth metadata construct defininga plurality of marketing campaigns, each marketing campaign of theplurality of marketing campaigns associated with a correspondingselection of data processes of the plurality of data processes, and eachmarketing campaign of the plurality of marketing campaigns associatedwith a corresponding party of a plurality of parties.

The exemplary embodiment may also include a tenth metadata construct ofthe plurality of metadata constructs, the tenth metadata constructadapted to use the plurality of selectable attributes for a dimensioningand aggregating data for storage in the multidimensional data structure.

The exemplary embodiment may also include an eleventh program constructadapted to determine a plurality of available variations of at least oneof the following: selectable data processes, selectable data attributes,or selectable data segmentations; a twelfth program construct adapted toprovide data modeling and data analysis using the plurality ofmodifiable parameters, the plurality of selectable input data sources,and the plurality of selectable attributes; and a thirteenth programconstruct adapted to provide at least one selection recommendation forthe plurality of modifiable parameters, the plurality of selectableinput data sources, or the plurality of selectable attributes.

In yet another exemplary embodiment, the present invention provides atangible medium storing computer readable software for programming aparallel and asynchronous database architecture and data processingsystem for marketing campaign management and analysis. The tangiblemedium storing computer readable software comprises a first programconstruct of a plurality of program constructs, the first programconstruct comprising a plurality of modifiable parameters adapted to bemodifiable during data processing, the first program construct furtherdefining a plurality of selectable input data sources and a plurality ofselectable attributes; a second program construct of the plurality ofprogram constructs, the second program construct having a correspondingmetadata linkage to the first program construct, the second programconstruct defining a plurality of data structures for a plurality ofdata repositories of the computer system, the plurality of datastructures comprising at least one relational data structure and atleast one multidimensional data structure; a third program construct ofthe plurality of program constructs, the second program construct havingcorresponding metadata linkages to the first and second programconstructs, the third program construct adapted to provide for executionof a plurality of data processes using the plurality of modifiableparameters, the plurality of input data sources and the plurality ofselectable attributes, the third program construct further adapted tostore a plurality of results of the plurality of data processes in theplurality of data structures; and a fourth program construct of theplurality of program constructs, the fourth program construct havingcorresponding metadata linkages to the first, second and third programconstructs, the fourth program construct adapted to use the plurality ofselectable attributes for a dimensioning and aggregating data forstorage in the multidimensional data structure.

The plurality of data processes may also be adapted to execute inparallel and asynchronously using a plurality of data processing nodesof the data processing system. The exemplary embodiment may also providefor the storage in a memory or in a database of a correspondingplurality of interim results from the asynchronous and parallelexecution of the plurality of data processes, which may be as acorresponding set of unique identifiers.

In yet another exemplary embodiment, the present invention provides atangible medium storing computer readable program instructions forexecution by a computer system for marketing campaign management andanalysis. The tangible medium storing computer readable programinstructions comprises: a first program module defining a plurality ofmodifiable parameters adapted to be modifiable during data processingand a plurality of selectable attributes; a second program moduledefining a plurality of selectable input data sources, the plurality ofselectable input data sources comprising at least one consumer databaseand a plurality of consumer lists of individuals and households; a thirdprogram module defining a plurality of data structures for a pluralityof data repositories of the computer system, the plurality of datastructures comprising a first relational data structure, a secondrelational data structure storing a subset of information from the firstrelational data structure, and at least one multidimensional datastructure; a fourth program module adapted to execute a plurality ofmarketing campaign management data processes using the plurality ofmodifiable parameters, the plurality of selectable input data sourcesand the plurality of selectable attributes, the fourth program constructfurther adapted to perform a first plurality of data assembly processesand store assembled data in the first relational data structure, and toperform a second plurality of data reduction processes on the assembleddata and store reduced data in the second relational data structure; afifth program module adapted to use the plurality of selectableattributes for dimensioning and aggregation of the reduced data forstorage in the multidimensional data structure; a sixth program moduleto correlate invocation of the plurality of marketing campaignmanagement data processes with accounting functions and resourceutilization; and a seventh program module to track selection of theplurality of consumer lists.

In yet another exemplary embodiment, the present invention provides atangible medium storing a plurality of metadata modules for execution bya computer system for marketing campaign management and analysis, thetangible medium comprising: a first metadata module of a plurality ofmetadata modules, the first metadata module comprising a plurality ofmodifiable parameters adapted to be modifiable during data processing; asecond metadata module of the plurality of metadata modules, the secondmetadata module defining a plurality of selectable input data sourcescomprising at least one consumer database and a plurality of consumerlists; a third metadata module of the plurality of metadata modules, thethird metadata module defining a plurality of data structures for aplurality of data repositories of the data processing system, theplurality of data structures comprising a first relational datastructure storing a first subset of information, a second relationaldata structure storing a second subset of information from the firstrelational data structure, and at least one multidimensional datastructure storing a third subset of information from the secondrelational data structure which has been dimensioned and aggregated; afourth metadata module of the plurality of metadata modules, the fourthmetadata module having corresponding metadata linkages to the first,second and third metadata modules, the fourth metadata module adapted toexecute a plurality of data assembly processes using the plurality ofselectable input data sources and the plurality of modifiable parametersto form the first subset of information, the fourth metadata modulefurther adapted to execute a plurality of data reduction processes usingplurality of modifiable parameters to form the second subset ofinformation; a fifth metadata module of the plurality of metadatamodules, the fifth metadata module adapted to dimension and aggregatethe second subset of information to form the third subset ofinformation, using a plurality of selectable attributes; a sixthmetadata module of the plurality of metadata modules, the sixth metadatamodule adapted to perform a plurality of set operation on the thirdsubset of information using the plurality of selectable attributes; aseventh metadata module of the plurality of metadata modules, theseventh metadata module adapted to utilize the multidimensional datastructure to process a query and to generate a response to the query;and an eighth metadata module of the plurality of metadata modules, theeighth metadata module adapted to provide at least one selectionrecommendation for the plurality of modifiable parameters, the pluralityof selectable input data sources, or the plurality of selectableattributes.

In yet another exemplary embodiment, the present invention providestangible medium storing computer readable software for programming aparallel and asynchronous database architecture and data processingsystem for marketing campaign management and analysis. The tangiblemedium storing computer readable software comprises: a first programconstruct of a plurality of program constructs, the first programconstruct comprising a plurality of modifiable parameters adapted to bemodifiable during data processing; a second program construct of aplurality of program constructs, the second program construct having acorresponding metadata linkage to the first program construct, thesecond program construct adapted to obtain a first modifiable parameterof the plurality of modifiable parameters from the first programconstruct, the second program construct further adapted to provide forexecution of a first data process of a plurality of data processes usingthe first modifiable parameter; a third program construct of theplurality of program constructs, the third program construct having asecond metadata linkage to the first program construct, the thirdprogram construct adapted to obtain a second modifiable parameter of theplurality of modifiable parameters from the first program construct, thethird program construct adapted to execute a multithreaded data processof the plurality of data processes, in parallel and asynchronously usingthe second modifiable parameter, through a plurality of third metadatalinkages to a subset of program constructs of the plurality of programconstructs, wherein a selected program construct of the subset ofprogram constructs is further adapted to perform a set intersectionoperation on a corresponding plurality of sets of unique identifiersresulting from the asynchronous and parallel execution of themultithreaded process; and wherein a first subset of modifiableparameters of the plurality of modifiable parameters define a pluralityof selectable input data sources, a second subset of modifiableparameters of the plurality of modifiable parameters define a pluralityof selectable attributes, and wherein the plurality of data processescomprise at least two of the following data processes: a data assemblyprocess, a data reduction process, a data aggregation process, or a datafulfillment process.

These and additional embodiments are discussed in greater detail below.Numerous other advantages and features of the present invention willbecome readily apparent from the following detailed description of theinvention and the embodiments thereof, from the claims and from theaccompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

The objects, features and advantages of the present invention will bemore readily appreciated upon reference to the following disclosure whenconsidered in conjunction with the accompanying drawings and exampleswhich form a portion of the specification, in which:

FIG. 1 (or “FIG. 1”) is a block diagram illustrating exemplary systeminteraction, metadata and process flows of the various exemplary systemand method embodiments in accordance with the teachings of the presentinvention.

FIG. 2 (or “FIG. 2”) is a block diagram illustrating a first exemplarysystem embodiment in accordance with the teachings of the presentinvention.

FIG. 3 (or “FIG. 3”) is a block diagram illustrating a second exemplarysystem embodiment in accordance with the teachings of the presentinvention.

FIG. 4 (or “FIG. 4”) is a block diagram illustrating a third exemplarysystem embodiment in accordance with the teachings of the presentinvention.

FIG. 5, divided into FIGS. 5A and 5B (or “FIG. 5A and FIG. 5B”), is aflow diagram illustrating an exemplary method embodiment in accordancewith the teachings of the present invention.

FIG. 6 (or “FIG. 6”) is a block diagram illustrating an exemplarylinkage repository embodiment in accordance with the teachings of thepresent invention.

FIG. 7 (or “FIG. 7”) is a diagram illustrating an exemplary metadataembodiment for an exemplary system embodiment in accordance with theteachings of the present invention.

FIG. 8 (or “FIG. 8”) is a diagram illustrating in greater detailexemplary metadata module embodiments and metadata linkages for anexemplary system embodiment in accordance with the teachings of thepresent invention.

FIG. 9 (or “FIG. 9”), is a flow diagram illustrating an exemplary methodembodiment for set determination for modified-OLAP functionality inaccordance with the teachings of the present invention.

FIG. 10 (or “FIG. 10”) is a block diagram of fourth system embodimenthaving one or more application specific integrated circuits (ASICs) inaccordance with the teachings of the present invention.

DETAILED DESCRIPTION OF THE EXEMPLARY EMBODIMENTS

While the present invention is susceptible of embodiment in manydifferent forms, there are shown in the drawings and will be describedherein in detail specific examples and embodiments thereof, with theunderstanding that the present disclosure is to be considered as anexemplification of the principles of the invention and is not intendedto limit the invention to the specific examples and embodimentsillustrated.

Referring now to the drawings wherein like reference numerals are usedto identify identical components in the various views, FIG. 1 is a blockdiagram illustrating exemplary system interaction, metadata and processflows of the various exemplary system and method embodiments inaccordance with the teachings of the present invention. The varioussystem 100, 200, 300 embodiments are discussed in greater detail withreference to FIGS. 2-4 and 10.

FIG. 1 illustrates several truly novel features of the presentinvention. First, the exemplary systems and processes utilize a new formof metadata defining both processes and data structures (such asentities, tables, relationships). The inventive metadata is furthercomprised of a plurality of parameterized metadata modules and metadatalinkages, providing for both dynamic and static parameters, allowing thevarious systems and processes to be fully customizable, fullystandardized, and everything in between. Second, the exemplary systemarchitecture provides for distributed processes which can be performedin parallel and asynchronously, providing for high speed data processingand effective and valuable decoupling of the provision of output datafrom the provision of input data. Third, the exemplary systems andprocesses provide for user interactivity, with a user enabled to provideinput, and make selections and decisions, throughout the extensive dataprocessing, and to modify previous decisions. Furthermore, suchselections and decisions may be made of the basis of actual dataanalysis and segmentation, provided through real-time queries andresponses, unavailable in the prior art. Lastly, the exemplary systemsand processes allow for the introduction of artificial intelligenceprocessing, with an analytic recommendation processor (“ARP”) (alsoreferred to as a “virtual information consultant”) provided to aid inthe various user selections and decisions, also based on actual dataanalysis and segmentation.

Referring to FIG. 1, a plurality of metadata modules 50 are utilized todefine and specify system processes, flows, versions, parameters(modifiable/dynamic and unmodifiable/static), entities, attributes,users/clients, hardware resources, data sources, data repositories (datarelationships and data structures), and all other features utilized inthe various exemplary systems. The metadata modules 50 areinterconnected, illustrated as interconnections 55, with theinterconnections referred to as “metadata linkages”, to distinguishother forms of linkage utilized with the various embodiments of theinvention. The various metadata linkages may occur in a variety of ways,such as through shared or otherwise common or communicated parameters,common (shared) attributes, common or communicated process definitions,and common or communicated process elements or flows, for example. Thevarious parameters (dynamic and static) are illustrated as parameters95, as additional illustration of metadata linkage through parametersharing, such as parameters utilized across multiple processes definedby a corresponding plurality of metadata modules 50. As discussed below,the metadata modules 50 are typically implemented as a form of softwareor programming instructions, such as in various forms of SQL (SEQUEL),MY SQL, DB2, ORACLE, C, C++, Java, XML, and may be developed through oneor more corresponding and commercially available application programminginterfaces (“APIs”) provided by many service and software companies.

The systems 100, 200, 300 are comprised of a plurality of datarepositories and data structures and a plurality of processing nodes 30.As illustrated in FIG. 1, the various data repositories and datastructures include data sources 10, data warehouse 15, data mart 20, andmultidimensional data structure 25. The plurality of processing nodes 30perform a plurality of data processes, illustrated as data assemblyprocesses 60, data reduction processes 65, aggregate and dimensionprocesses 70, and a fulfillment process 75, and may do so in paralleland asynchronously, depending upon the selected embodiment. As aconsequence, any data output has been effectively decoupled from datainput, allowing tremendous versatility for rapid data processing, dataupdating, data analysis and modeling, and other data processes.

The various metadata modules 50 are illustrated as correspondinglymapping to the plurality of data processes, data sources/inputs and datarepositories and data structures, as the metadata modules 50 define theprocesses, provide for invocation of the processes, determine the datastructures to be utilized or created, provide for various neededparameters, attribute selection, data input selection, and othercustomizable features. As a consequence, the metadata of the presentinvention is parameterized, namely, providing for the use of variable ordynamic parameters, such as varying confidence levels, attributechoices, process selection, process ordering, and so on, in addition toproviding for static parameters. Through the parameterized metadata, thedata processing of the present invention may be provided in differentversions, different process flows, and may be completely customizable.In addition, through the use of default values, the data processing ofthe present invention may be completely standardized or fixed. Through acombination of default and selected values, the data processing of thepresent invention provides a complete panorama, from fully customizableto fully standardized and everything in between.

The parameters, attribute selection, process selection, processordering, selected dimensions for aggregation into the multidimensionaldata structure (25), and other customizable features, may be enteredinto the system through a user or control interface or terminal (135,175, 355), such as through a graphical user interface on a personalcomputer, coupled to the exemplary systems through a network, such as aLAN or the Internet. Through the user or control interface (135, 175,355), virtually all aspects of the data processing may be controlled,and controlled interactively and dynamically. For example, at thebeginning of the data processing, a user may make selections concerningwhich data sources and data inputs (10) will be utilized, whichprocesses (e.g., 60, 65, 70, 75) will be utilized, using whichparameters, and using which selected attributes (35). At other timesduring the data processing (indicated as a subsequent time), a user maymake other or additional selections concerning which processes,parameters and attributes will be utilized, and may also modify earlierselections and decisions (40).

As a significant consequence, important decisions may be made throughoutthe data processing, and may be made based on real-time data analytics,segmentation and modeling, such as by evaluating various “what if”scenarios using the actual data which has been generated. Thisintelligence is provided through the use of interactive and real-timequeries and responses (80), providing different analyses andsegmentation of the multidimensional data structure 25. In addition,various decision suggestions may be received by the user from theanalytic recommendation processor 90 based, for example, upon modelingand analytics (85), such as modeling from previous marketing campaignsand modeling from real-time analyses and segmentation of themultidimensional data structure 25.

The exemplary embodiments of the invention also provide for massivelyparallel and asynchronous data processing. Such processing is supportedby the plurality of metadata modules, which are also adapted to providefor execution of such multithreaded data processes, across multipleprocessors.

FIG. 2 is a block diagram illustrating a first exemplary systemembodiment 100 in accordance with the present invention, with exemplaryprocessing nodes 110 and 150, and with exemplary file and/or databasesystems 145, illustrated as file and/or database systems 145A, 145B,145C and 145D. Such one or more processing nodes 110, 150 and fileand/or database systems 145 may be programmed, adapted or configuredwith metadata, parameterized metadata, software, program or othercomputer instructions or configuration information to operate inaccordance with the teachings of the present invention.

The system 100 may also be examined in conjunction with FIGS. 3 and 4,which are block diagrams illustrating second exemplary system 200 andthird exemplary system 300 embodiments in accordance with the teachingsof the present invention, and which provide a different level of detailof the various system embodiments (100, 200 or 300) and the variousexemplary subsystems or nodes of the present invention. For example,referring to FIG. 3, depending upon the selected embodiment, the variousone or more mainframe process nodes 205, clustered database nodes 210,Unix application nodes 215, Windows® application nodes 220, networknodes 230, and control systems 260 may include, be comprised of, orotherwise be represented by processing nodes 110 and/or 150. Similarly,the various data storage systems of FIG. 3, such as the mainframe filesystems 240, partitioned database storage 245, Unix file systems 250,and Windows® file systems 255 may include, be comprised of, or otherwisebe represented by database and/or file systems 145.

Correspondingly, referring to FIG. 4, depending upon the selectedembodiment, the various processing elements such as data assemblyprocessor 315, data reduction processor 325, aggregation processor 335,fulfillment processor 345, analytic recommendation processor 350, anddata processing selection and planning node 355 may include, becomprised of, or otherwise be represented by processing nodes 110 and/or150. Similarly, the various data storage systems of FIG. 4, such as thedatabase, file and media systems 310, data warehouse 320, data mart 330,and analytics and segmentation processing data structure (data cube) 340may include, be comprised of, or otherwise be represented by databaseand/or file systems 145. As a consequence, the methodology and systemsof the present invention should be understood by those of skill in theart to be adapted to or capable of implementation using any form or typeof hardware, currently known or which may become available in thefuture, with all such physical implementations considered equivalent andwithin the scope of the present invention. For example, exemplaryprocesses which may currently operate on mainframe process nodes 205,such as legacy software applications, may be migrated or ported to otherprocessing nodes, such as clustered database nodes 210, Unix applicationnodes 215, Windows® application nodes 220.

The various exemplary embodiments of the invention will be explained indetail with regard to a particular voluminous, computationally and dataintensive process, such as marketing campaign management, segmentationand analysis. Those of skill in the art will recognize that the datastructures, systems, principles, and procedures discussed below areequally applicable to other typically voluminous, difficult orintractable data processing tasks, with additional applications todiverse fields such as medical and scientific research, speech andsignal processing, and so on, with additional illustrated examplesdiscussed below.

As illustrated in FIG. 2, the processing node 110 is a computer systemembodiment (e.g., a mainframe computer, a personal computer, a computerworkstation, or any other computer having processing power sufficient toexecute the exemplary method discussed below), comprising an input andoutput (I/O) interface 105, one or more processors 115, and a memory 120storing one or more databases or file systems (individually andcollectively referred to as data repositories) 145A. The memory 120 maybe external, such as an external magnetic disk, tape, or optical drive,or may be internal, such as random access memory, cache memory, and soon. The processing node 150, such as an open or network system,comprises one or more data repositories (or databases) 145B (alsoembodied in a form of memory, discussed below), a database managementserver 140, and/or an application server 125. Additional file and/ordatabase systems 145C and 145D are directly coupled to data/messagingnetwork 235, for access from any processing node 110, 150, or othercomputer 175 (or other type of user interface) or user/control node 135.A “data repository”, “database”, and “data warehouse”, as used herein,are considered interchangeable, and may be relational, object-oriented,object-relational, or use files or flat files, or any combinations ofthe above. Databases 145A, 145B, 145C, and 145D are instantiations ofone or more databases, data warehouses, data marts, or data cubes,discussed in greater detail below. Reference to a “data cube” will beunderstood to mean a particular multidimensional instantiation of such adatabase 145, adapted in advance through an aggregation process(discussed below) for particularly responsive query processing, andwhich typically will have a plurality of different views which may bematerialized (e.g., a lattice of views).

In the exemplary embodiments of processing node 150, the databasemanagement server 140 and the application server 125 may be implementedtogether, such as implemented within the application server 125. Eitheror both of the database management server 140 and the application server125 are connected or coupled (or couplable) to the one or more datarepositories (or databases) 145B, for full duplex communication, such asfor database queries, database file or record transfers, databaseupdates, and other forms of database communication. In the processingnode 150, the database management server 140 and/or the applicationserver 125 perform the methodology of the invention utilizing acorrespondingly adapted, programmed or configured processor as discussedbelow (not separately illustrated), such as a processor 115 illustratedfor processing node 110, in conjunction with one or more databases 145(such as one or more databases 145B).

The databases 145A, 145B, 145C, and 145D may be ODBC-compliant (OpenDatabase Connectivity), although this is not required for the presentinvention, or may be mainframe or proprietary. The processing node 110,processing node 150 and databases 145 are coupled to one another througha data and messaging network 235, typically implemented as one or moredata and messaging busses, and may also be coupled to or may be part ofa local area network (“LAN”) 130 or a wide area network (“WAN”) 132,such as for full duplex communication with a plurality of computers (orother terminals) 175, also for database queries, database file or recordtransfers, database updates, and other forms of database communication.The LAN 130, WAN 132, and data and messaging network 235 communicationcapability provides for local and remote access to the processing node110, processing node 150 and databases 145, such as for large filetransfers (or any batch processing, to the extent needed), and forprocesses such as control, and data selection and planning, throughvarious computers 175 and one or more user/control nodes 135. Inaddition, the systems (100, 200, or 300) may also be directly accessiblethrough input/output (“I/O”) 185, such as for loading of records (e.g.,magnetic tape records, disk records, or other media).

The various systems (100, 200, or 300) may also be included within orcoupled to a larger data communication network 180, such as throughnetwork (or web) server 160, for full duplex communication with remotedevices, such as a remote Internet or other network server 170 andremote computer (or other terminal) 175. In addition to local accessthrough one or more local computers 175 and user/control nodes 135, suchremote communication capability provides for the systems (100, 200, or300) to be accessible for on-line functionality, discussed in greaterdetail below, such as for web-based access, using any current or futureprotocols, such as hypertext transfer protocol (“HTTP”), various emailand file transfer protocols (e.g., SMTP, FTP), Transmission ControlProtocol (“TCP”), Internet Protocol (“IP”), or any other forms ofcommunication for data, voice or multimedia, for user input, selection,evaluation, reporting, and so on, discussed below. The user/controlnodes 135 may be implemented as known or becomes known in the art, suchas through a personal computer, a client-server embodiment, a dataterminal, for example, typically having a monitor for display ofinformation, such as a video display or computer monitor, and havingdata input means, such as a keyboard, pointing device (e.g., mouse,trackball, touch pad), joystick, scanner, touch screen, microphone,voice recognition system, keypad, and so on.

The data repository (or database) 145, illustrated as databases 145A,145 B, 145C, and 145D, may be embodied in any number of forms, includingwithin any computer or other machine-readable data storage medium,memory device or other storage or communication device for storage orcommunication of information such as computer-readable instructions,data structures, program modules or other data, currently known or whichbecomes available in the future, including, but not limited to, amagnetic hard drive, an optical drive, a magnetic disk or tape drive, ahard disk drive, other machine-readable storage or memory media such asa floppy disk, a CDROM, a CD-RW, digital versatile disk (DVD) or otheroptical memory, a memory integrated circuit (“IC”), or memory portion ofan integrated circuit (such as the resident memory within a processorIC), whether volatile or non-volatile, whether removable ornon-removable, including without limitation RAM, FLASH, DRAM, SDRAM,SRAM, MRAM, FeRAM, ROM, EPROM or E²PROM, or any other type of memory,storage medium, or data storage apparatus or circuit, which is known orwhich becomes known, depending upon the selected embodiment. Inaddition, such computer readable media includes any form ofcommunication media which embodies computer readable instructions, datastructures, program modules or other data in a data signal or modulatedsignal, such as an electromagnetic or optical carrier wave or othertransport mechanism, including any information delivery media, which mayencode data or other information in a signal, wired or wirelessly,including electromagnetic, optical, acoustic, RF or infrared signals,and so on.

The network 235 may be any type of bus or network structure or medium,using any selected architecture. By way of example and not limitation,such architectures include Industry Standard Architecture (ISA) bus,Enhanced ISA (EISA) bus, Micro Channel Architecture (MCA) bus,Peripheral Component Interconnect (PCI) bus, SAN bus, or any othercommunication or signaling medium, such as Ethernet, ISDN, T1,satellite, wireless, and so on. In the processing node 110, the I/Ointerface 105 may be implemented as known or may become known in theart, and as discussed with reference to FIG. 10, to provide datacommunication between the processor 115 and the network 235, using anyapplicable standard (e.g., one of the various PCI, USB or Ethernetstandards, for example and without limitation).

The processing node 110 and processing node 150 further include one ormore processors, such as processor 115 illustrated for processing node110, and the other processors referred to in the other systemembodiments discussed below. As the term processor is used herein, theseimplementations may include use of a single integrated circuit (“IC”),or may include use of a plurality of integrated circuits or othercomponents connected, arranged or grouped together, such asmicroprocessors, digital signal processors (“DSPs”), parallelprocessors, multiple core processors, custom ICs, application specificintegrated circuits (“ASICs”), field programmable gate arrays (“FPGAs”),adaptive computing ICs, associated memory (such as RAM, DRAM and ROM),and other ICs and components. As a consequence, as used herein, the termprocessor should be understood to equivalently mean and include a singleIC, or arrangement of custom ICs, ASICs, processors, microprocessors,controllers, FPGAs, adaptive computing ICs, or some other grouping ofintegrated circuits which perform the functions discussed below, withassociated memory, such as microprocessor memory or additional RAM,DRAM, SDRAM, SRAM, MRAM, ROM, EPROM or E²PROM. A processor (such asprocessor 115), with its associated memory, may be adapted or configured(via programming, FPGA interconnection, or hard-wiring) to perform themethodology of the invention, as discussed above and as furtherdiscussed below. For example, the methodology may be programmed andstored, in a processor with its associated memory (and/or memory 120)and other equivalent components, as a set of program instructions (orequivalent configuration or other program) for subsequent execution whenthe processor is operative (i.e., powered on and functioning).Equivalently, when the processing node 110 and processing node 150 mayimplemented in whole or part as FPGAs, custom ICs and/or ASICs, theFPGAs, custom ICs or ASICs also may be designed, configured and/orhard-wired to implement the methodology of the invention. For example,the processing node 110 and processing node 150 may implemented as anarrangement of microprocessors, DSPs and/or ASICs, collectively referredto as a “processor”, which are respectively programmed, designed,adapted or configured to implement the methodology of the invention, inconjunction with one or more databases 145.

An exemplary one or more data repositories (or databases) 145, inaccordance with the present invention, contains public data, privatedata, survey data, limited access data, historical data, synthetic data,response data, intermediate result data, retained relationships (sets)data, etc., which themselves may be in the form of one or more relationsand/or databases (as illustrated). In addition, exemplary databases 145may contain the results of the method of the invention, as a datastructure, discussed below. As mentioned above, “data repository” asused herein, is considered interchangeable with “database” and “datawarehouse”, and may be relational, object-oriented, orobject-relational, or utilize any other database structure, inaccordance with a selected embodiment. The database 145 may beintegrated, namely, that the information resides within a singular,co-located or otherwise centralized database structure or schema, or maybe a distributed database, with information distributed between andamong a plurality of databases, some of which may be remotely locatedfrom the other databases. From another point of view, the database 145may be considered integrated in that a plurality of different tables ortypes of tables, objects or relations are included within the database145. (Use of any type of data repository, whether an integrateddatabase, a non-integrated database, or any otherwise distributed ornon-distributed database structures or schemas, are within the scope ofthe present invention. Unless otherwise specified, while referred to astables, it should be understood that any such tables are to be construedbroadly, to mean and include relations, objects, object relations,multidimensional relations, cubes, stars, snowflakes, flat files, orother similar or equivalent database constructs.)

The plurality of processing nodes (110, 150, 205, 210, 215, 220) and theother processors and processing nodes referred to as a data assemblyprocessor 315, data reduction processor 325, aggregation processor 335,fulfillment processor 345, analytic recommendation processor 350, may beimplemented using any form of server, computer, workstation, mainframe,or other computational device as known or may become known in the art,such as a server or other computing device having a processor,microprocessor, controller, digital signal processor (“DSP”), adaptivecomputing circuit, or other integrated circuit adapted, programmed orconfigured to perform the methodology of the present invention, such asa processor 115, as mentioned above. The system of the present inventiongenerally comprises a memory storing a data repository (or database) 100and one or more of such processors. The processor is programmed, usingdata processing software and using other software and data structuresdefined by the parameterized metadata of the invention, for example, toperform the methodology of the present invention. As a consequence, thesystem and method of the present invention may be embodied as softwarewhich provides such programming, such as a set of instructions and/ormetadata embodied within a computer readable medium, discussed above.

More generally, the system, methods, metadata and programs of thepresent invention may be embodied in any number of forms, such as withinany type of computer, within a workstation, within an application serversuch as application server 125, within a database management server 140,within a computer network, within an adaptive computing device, orwithin any other form of computing or other system used to create orcontain source code, including the various processors and computerreadable media mentioned above. Such source code further may be compiledinto some form of instructions or object code (including assemblylanguage instructions or configuration information). The metadata,software or source code of the present invention may be embodied as anytype of source code, such as SQL and its variations (e.g., SQL 99 orproprietary versions of SQL), DB2, XML, C, C++, Java, Oracle, or anyother type of programming language which performs the functionalitydiscussed herein, including various hardware definition languages (e.g.,Verilog, HDL), when embodied as an ASIC. As a consequence, a“construct”, “program construct”, “software construct” or “software”, asused herein, means and refers to any programming language, of any kind,with any syntax or signatures, which provides or can be interpreted toprovide the associated functionality or methodology specified (wheninstantiated or loaded into a processor or computer and executed,including any of the plurality of processing nodes (110, 150, 205, 210,215, 220) or other processors and processing nodes referred to as a dataassembly processor 315, data reduction processor 325, aggregationprocessor 335, analytic recommendation processor 350, or fulfillmentprocessor 345, for example).

The metadata, software or other source code of the present invention andany resulting bit file (object code or configuration bit sequence) maybe embodied within any tangible storage medium, such as within a memoryor storage device for use by a computer, a workstation, any othermachine-readable medium or form, or any other storage form or medium foruse in a computing system. Such storage medium, memory or other storagedevices may be any type of memory device, memory integrated circuit(“IC”), or memory portion of an integrated circuit (such as the residentmemory within a processor IC), including without limitation RAM, FLASH,DRAM, SRAM, MRAM, FeRAM, ROM, EPROM or E²PROM, or any other type ofmemory, storage medium, or data storage apparatus or circuit, dependingupon the selected embodiment. For example, without limitation, atangible medium storing computer readable metadata or software, or othermachine-readable medium, may include a floppy disk, a CDROM, a CD-RW, amagnetic hard drive, an optical drive, a quantum computing storagemedium or device, a transmitted electromagnetic signal (e.g., used ininternet downloading), or any other type of data storage apparatus ormedium, as mentioned above.

In addition, the various structures and components referred to in FIG. 2are applicable to the other system embodiments discussed below and, inthe interests of brevity, will not be repeated. Similarly, the variousstructures and components referred to in FIGS. 3 and 4 are applicable tothe other system embodiments and, in the interests of brevity, will notbe repeated. As a consequence, those of skill in the art will recognizethat the various systems illustrated below may be embodied in the widevariety of structures and components discussed above and below, whichare all within the scope of the present invention. As a furtherconsequence, reference to any of the various systems (100, 200, or 300)should be understood to mean and include reference to any of the othersystems (100, 200, or 300), unless otherwise specified.

Referring to FIG. 3, the exemplary system 200 provides a serviceoriented architecture (“SOA”), and includes a plurality of processingnodes, control systems, and storage or repository structures which areadapted to communicate with each other, and with other networks (asillustrated in FIG. 2 and not separately illustrated in FIG. 3), usingdata and messaging network 235, which may be implemented, for example,as various data and messaging busses or lines, or as other types ofcommunication structures currently known or which becomes known. Inexemplary embodiments, the data and messaging network 235 is implementedas two types of busses, one for large or high bandwidth data transfers,and another for messaging and comparatively smaller (lower bandwidth)data transfers, between and among the various nodes and storage devices.

Continuing to refer to FIG. 3, the various nodes include, withoutlimitation, one or more mainframe processing nodes 205 (such asmainframe computers), one or more clustered database nodes 210, andother computing nodes with various operating systems, such as one ormore Unix application nodes 215 and one or more Windows® applicationnodes 220. Communication nodes are also provided, using one or morenetwork nodes 230, such as for LAN, WAN, Internet, and other networkcommunications. For example, network nodes 230 generally include variousapplication and web servers discussed above, and as illustrated in FIG.4, for local or remote communication with users and operators of thesystem 200. One or more control systems 260 are also provided, whichsupply metadata (from data repository 265) defining the various system200 applications, processes and services. The metadata of repository 265is typically for static (or non-parameterized metadata), utilized forall of the processes discussed below. (Parameterized metadata, discussedbelow, is utilized to provide system variations and customizations, on aclient- or user-specified basis.) The one or more control systems 260typically further provide orchestration of the various applications,processes and services, such as timing, invocation, and ordering, andwhich typically provide other services, such as system status monitoringand management, billing, accounting, and so on. Not separatelyillustrated in FIG. 3, the system 200 may also include one or moreinterfaces (such as I/O 105 or 185) or drives for input of data onvarious media, such as tape, disk, CD, etc.

The various data storage systems generally may include withoutlimitation, as needed depending upon and corresponding to the variousprocessing and application nodes selected, one or more mainframe filesystems 240, partitioned database storage 245, one or more Unix (orother operating mode) file systems 240, and one or more Windows® filesystems 240. These storage elements may be implemented in a stored areanetwork (“SAN”) environment, in selected embodiments, with allprocessing nodes having data access through a dedicated SAN network, forexample, and with messaging and control provided on a separate networkor bus, which together form the network 235, as an exemplary embodiment.The partitioned database storage 245 may be embodied as one or moredatabases 145 discussed above, and includes without limitation a generalinformation and attribute data repository 270 (such as an ExperianINSOURCE® database, Z24® database, Circbase® database), a linkagerepository 275A, a retained relationship linkage repository 275B, ametadata repository 280 such as for parameterized metadata (as may bespecified or selected by a client or other data user), one or moreprivate repositories 285 (such as for client data or other types ofprivate or privileged data, with a separate private repository 285implemented for each client), one or more data warehouses 295, and oneor more data marts 290, illustrated as data marts 290A and 290B. Thefile systems 240, 250 and 255 may include any type of data storagesystems, including databases, flat files, web (e.g., HTML or XML) pages,documents, and so on. In the exemplary embodiments discussed below,these file systems (such as Unix (or other operating mode) file systems240 or Windows® file systems 240) may also include the data cube (OLAP)structures discussed below, such as one or more of the analytics andsegmentation processing data structures illustrated in FIG. 4.(Alternatively, the metadata of repository 265 and the metadatarepository 280 may be switched or reversed, with the metadata ofrepository 265 containing parameterized metadata, and the metadatarepository 280 containing static or non-parameterized metadata.) Thevarious file and database systems may be implemented as known or as maybecome known in the art, and all variations are within the scope of thepresent invention.

The various processing and application nodes (205, 215, 220) anddatabase nodes 210 provide the various processes (also referred to asservices) and applications discussed below, such as data reception andinput, data layout, data conversion and correction, data audits,matching and linkage assignment, data coding, data enhancement, datalinkage, data change detection and update services, data netdown, datasuppression, data prioritization, data output (push) applications (suchas automatic updates to client databases), variable formatting services,various fulfillment services (e.g., report generation, postal services,postal presort services, inkjet and other printing preparation services,shipping services, data extraction, email formatting, etc.). Theseservices may be provided in a wide variety of orders, with the storageof any interim results (e.g., in retained relationships linkagerepository 275B), using service orchestration provided by controlsystems 260.

The various processing and application nodes (205, 215, 220) anddatabase nodes 210 further provide user- or client-specified services,generally through a graphical user interface (“GUI”, not separatelyillustrated), such as via a computer 175 or user/control node 135,including data selection and processing planning, result reporting andanalysis, report rendering, metadata parameterization, selection andentry, aggregation selection, attribute selection, analytics andsegmentation processing, and web or other network hosting. In addition,the various processing and application nodes (205, 215, 220) anddatabase nodes 210 also provide artificial intelligence services, as ananalytic recommendation engine or processor, such as data modeling, dataaggregation, data consultation, data analysis, and user interaction(illustrated as analytic recommendation processor 350 in FIG. 4), andalso occasionally referred to herein as “virtual information consulting(or consultant)” services.

These various services and processes may be distributed between andamong the various processing and application nodes (205, 215, 220) anddatabase nodes 210 in a wide variety of equivalent ways, which may be afunction of existing legacy systems, legacy software, customizedsoftware, software licenses, and costs and expenses. For example,computationally intensive tasks may be performed using comparativelyinexpensive CPUs available on nodes 215 and 220, rather than using thepotentially more expensive nodes 205 and 210. Also for example, certainprocesses which are invoked less frequently may be maintained on legacysystems, with other, more frequently executing processes ported to aplurality of nodes 215 and 220, for increased distributed and parallelexecution. More specifically, such distributed processing, inconjunction with the linked and modular metadata of the presentinvention, also facilitates the capability of the systems (100, 200 and300) for parallel and asynchronous execution of the plurality ofprocesses of the present invention.

The linkage repository 275A stores and maintains a database having aplurality of unique and persistent identifiers (such as persistentidentification numbers or “PINs”), with each PIN assigned to orassociated with each individual, household, business, or other entity orobject contained in the various other repositories, such as repository270. In addition, the (entity) linkage repository 275A also maintainsrelationships between or among each such entity and variouscorresponding data entries or attributes, referred to generally as“entity linkage” or simply “linkage”, to distinguish the metadatalinkage of the exemplary embodiments. For example, an individualassigned PIN 1234 may be matched to hundreds or thousands of otherrecords, including with name and address variations, and with otherattributes such as age, gender, income, social security number, vehicleownership, marital status, spouse name and address, children names andaddresses, household identification, home ownership, purchasinghistories, marketing responses, employment histories, and so on, alongwith any other attributes which may be maintained in repository 270,data warehouse 295, or in one or more private repositories 285. As aconsequence, selection of a given PIN may be utilized to provide linkageand immediate access to all of the other maintained and relatedattribute information for the selected entity. For example, as discussedbelow with reference to FIG. 5, once a PIN is selected for storage indata warehouse 320, the entire history associated with each such PIN isimmediately available, including all current and past information andrelationships. In addition, as data is being processed, intermediate orinterim results may be stored in retained relationship linkagerepository 275B, as a plurality of PINs, such as separate sets of PINsresulting from different parameters (or versions) utilized in datasuppression, data netdown, and data prioritization, for example.

As records or other data are input into the various systems (e.g., 100,200 or 300), they may be matched to existing records using such PINs.Where the input records do not include corresponding PINs, they may bematched using matching logic as known in the art, and when a match isfound, assigned the corresponding PIN. When a match is not found, therecord is typically assigned a new PIN. The various PINs and otherlinkage data are maintained in linkage repository 275A. Such matchingand PIN assignment is described in various publications, such as arelated U.S. patent application Ser. No. 10/651,469, entitled “System,Method And Software For Providing Persistent Business EntityIdentification And Linking Business Entity Information In An IntegratedData Repository”, filed Aug. 29, 2003, inventors John L. Skurtovich, Jr.et al., which further claims priority to provisional application Ser.No. 60/474,350, filed May 29, 2003, which is commonly assigned herewith,and all of which are incorporated by reference herein, with priorityclaimed for all commonly disclosed subject matter.

Another novel feature of the linkage repository 275A is a layeredimplementation, based upon comparative confidence levels, as illustratedin FIG. 6. Referring to FIG. 6, at the deepest level is validatedinformation 505, having a comparatively high confidence level (“A”),followed by trusted but non-validated information 510 having a somewhatlower confidence level (“B”) in comparison to confidence level “A”,followed by other information 515 having a somewhat lower confidencelevel (“C”) in comparison to confidence levels “B” and “A”, and furtherhaving one or more private linkage data stores 520, having anyconfidence level as determined by a given client or other data source,and the retained relationship linkage repository 275B. Each of theseconfidence levels provides a different “view” into the correspondingdata, and may be selected by corresponding parameters reflectingcorresponding confidence levels, such as through user input, clientspecifications, default values, etc. For example, when coarser-grainedanalysis will be utilized, such as at a household rather than individuallevel, less confidence may be required for matching, allowing use oftrusted but non-validated information, rather than the strictervalidated information. In addition, with higher confidence levelscreated, for example, through additional confirmatory or validatinginformation, information may migrate or propagate inward, toward trustedand toward validated information, all through associated PINinformation.

In accordance with the exemplary embodiments, these different confidencelevels of linkage, representing validated or non-validated information,can be applied independently to any of the various attributes andentities, including individuals, households, and addresses, for example.

Instead of being limited to the one-to-one linkage of the prior art, inwhich one entity is linked to one identifier, another novel linkagefeature of the present invention provides “many-to-many”, “one-to-many”or “many-to-one” linkages, also using assigned and unique persistentidentifiers or keys. These additional linkages are also maintained, asretained relationships, allowing the virtually instant accessibility ofall such linked information. For example, such many-to-many linkages mayassociate multiple names to multiple addresses, such as two names of acouple, linked to multiple addresses, indicative of a household livingarrangement. Similarly, a many-to-many linkage may associate multiplenames with multiple mobile telephone numbers, such as available in afamily or group mobile communications plan.

Also contrary to the prior art, the various systems (100, 200 or 300)maintain such linkage utilizing normalized relational data structures,without limiting the number of stored variations, despite the increasein processing complexity which has prevented prior art systems fromusing such normalized data structures. The systems (100, 200 or 300)provide such additional linkage by providing for distributed dataprocessing, such as by using a plurality of processing and applicationnodes (205, 215, 220) and database nodes 210, where match processing mayoccur in parallel and asynchronously on multiple candidate sets. As aconsequence, multiple pieces of data, for example, may be linked tomultiple entities at the same time.

Another novel feature of the present invention, and as discussed ingreater detail below, the various systems (100, 200 or 300) leverage thelinkage system by retaining and storing interim results as “retainedrelationships” (in the retained relationship linkage repository 275B),that is, results which are produced in a first subprocess and which canbe utilized as input into a second subprocess. For example, a set ofPINs or other linkage identifiers, produced as result of a firstsubprocess, are stored separately from the final results produced by thesecond subprocess. As a consequence, such retained relationships allowsthe separation of a single process into a plurality of decoupledsubprocesses, which may then be processed in parallel or in differentorders, for example, rather than serially and in a predetermined order.In addition, such enabling of process separation and decoupling providesfor additional advantages, such as the ability to separately customizethe decoupled processes.

FIG. 4 is a block diagram illustrating a third exemplary system 300embodiment in accordance with the teachings of the present invention.The operation of the system 300 is described in detail below withreference to FIG. 5. Referring to FIG. 4, the system 300 includesrelational data structures such as database, file and media systems 310,data warehouse 320, data mart 330, and OLAP (on-line analyticalprocessing) data structures, such as analytics and segmentationprocessing data structure (data cube) 340 (as database (or file)structures typically embodied using data storage elements such asmainframe file systems 240, partitioned database storage 245, Unix (orother operating mode) file systems 240, and Windows® file systems 240).For example and without limitation, in selected embodiments, analyticsand segmentation processing data structure (data cube) 340 is embodiedusing Windows® file systems 240, while data mart 330 and data warehouse320, respectively, are embodied as one of the data marts 290 and datawarehouses 295 in partitioned database storage 245. The system 300further includes data assembly processor 315, data reduction processor325, data aggregation processor 335, fulfillment processor 345, dataprocessing selection and planning node 355, control systems 260, andanalytic recommendation processor (artificial intelligence (A/I) node)350, as processors typically embodied using processing and applicationnodes (205, 215, 220) and database nodes 210, or more generally, asprocessing nodes 110 or 150 (with any incorporated processors 115).

The other illustrated components, such as user/control node 135, LAN130, WAN 132, network (web) server 160, network 180, and computers (oruser interfaces) 175, are utilized to provide user or client interactionand system control (such as input from a system monitor), and may beembodied as previously discussed with reference to FIG. 2. For example,the user/control node 135 or computers 175 (such as via LAN 130, WAN132, or direct connection to node 355) may be utilized to provideoperator, user or client access to the system 300, for activities suchas marketing campaign management, including data input selection,attribute selection, parameter selection (using data processingselection and planning node 355), and to provide interactivesegmentation and analysis (using analytics and segmentation processingdata structure (data cube) 340). The various illustrated components arecoupled to data/messaging network 235, for data and messagingtransmission and reception, as discussed with reference to FIG. 3.

The database and processing architecture of FIG. 4, as discussed ingreater detail below, provides tremendous advantages for processinglarge amounts of complicated data, such as in marketing campaignmanagement. First, data which tends to change comparatively slowly is:(1) selected, using operator, user, client and artificial intelligenceinput from data processing selection and planning node 355 and/oruser/control node 135; (2) input from database, file and media systems310; and then (3) assembled by data assembly processor 315 and stored indata warehouse 320, with changes and other updates as needed. Dataassembly processor 315 typically performs computationally intensiveprocesses, such as data conversion and correction, linkage, dataenhancement with additional fields or attributes, and other common dataprocessing. The data assembly processor 315 may be distributed among aplurality of processing nodes (110, 150, 205, 210, 215, 220), and dataassembly execution may occur in parallel and asynchronously, dependingupon the selected embodiment. As a consequence, comparatively staticinformation is warehoused and utilized to provide the data “universe”for selection and analysis, without these computationally intensive dataoperations needing to be repeatedly performed for subsequentmodifications (e.g., changes in attribute selection) or subsequentcreation of various permutations, combinations, selections,segmentations and analyses.

As not all of the information stored in data warehouse 320 will beutilized in a given or subsequent marketing campaign, but may beutilized in subsequent campaigns or other endeavors, to increasesubsequent processing and query response speeds, the information storedin data warehouse 320 will be reduced, using various processesimplemented by data reduction processor 325, with the reduced datastored in a data mart 330, typically implemented as a relationalstructure. This data reduction may be performed as needed or as desired,such as for other campaigns or other applications, and may result in aplurality of different data marts 330, each produced for the selectedapplication, with corresponding selected information. For example, datareduction processor 325 may implement data netdown, data suppression,and data prioritization for one or more selected marketing campaigns. Aparticularly novel feature of the present invention, as these datareduction processes are performed, the interim results are stored, as“retained relationships”, using corresponding PINs, in retainedrelationships linkage repository 275B. As a consequence, anyreprocessing using different reduction criteria does not requirecomplete reprocessing; instead, using the retained relationships, theprevious set of non-reduced data may be recovered essentiallyimmediately, and then reprocessed utilizing the new criteria. Inaddition, using the retained relationships, such processing andreprocessing may occur in a wide variety of orders, rather than aparticular predetermined order.

The resulting information stored in data mart 330 is then aggregated,using data aggregation processor 335, based on selected attributes ordimensions also input from the operator, client, other user from dataprocessing selection and planning node 355 or user/control node 135, orfrom analytic recommendation processor 350, and stored in analytics andsegmentation processing data structure (data cube) 340, to facilitatevirtually real-time segmentation, modeling, and analysis. Comparing thedata mart 330 and the analytics and segmentation processing datastructure (data cube) 340, the data mart 330 is typically notpre-aggregated and is a relational structure, and which may generallyhave a star or snowflake schema, for example and without limitation, andtypically contains additional information which may not be needed foranalysis, but may be useful for other processes, such as reportgeneration. The analytics and segmentation processing data structure 340is aggregated using selected attributes and dimensions; while typicallyreferred to as a “data cube”, it may be considerably moremultidimensional (i.e., more than 3 dimensions of a cube), and isadapted for real-time responsiveness to input queries. For example, aclient (through user interfaces or computers 175 or user/control node135) may submit various queries for segmentation on certain attributes,such as age or income, and using those results, determine whichindividuals (represented in the analytics and segmentation processingdata structure (data cube) 340) should receive marketing or promotionalmaterials. In addition, during this process, based upon responses topast campaigns, or based upon analytical modeling of the individuals,households, businesses or other entities represented in data mart 330,analytic recommendation processor 350 may also suggest certainsegmentations and analyses to provide an increased response rate for thecurrent marketing campaign.

In this interactive process, a selection of individuals, households orbusinesses is made, for use in a current marketing campaign. The PINs ofthese individuals, households or businesses are then provided tofulfillment processor 345, which performs various fulfillment processes,such as generation of name and address information, generation of emailaddresses, variable formatting, layout, inkjet services, postal presort,other postal services, and so on, as mentioned above and as discussed ingreater detail below.

FIG. 5 is a flow and block diagram illustrating an exemplary methodembodiment in accordance with the teachings of the present invention,and illustrates an exemplary use of the system 300 with reference tomarketing campaign management, execution and fulfillment. Referring toFIG. 5, the method begins, start step 400, with data selection andplanning, step 405, which may include input and suggestions fromanalytic recommendation processor 350, providing data intelligence, step410. For campaign management, step 400 typically includes designation ofdata repositories 412 and data sources 414, such as designation ofprivate repository 285, general repository 270, and consumer or businesslists to be provided as input and to be used for selection ofindividuals, households, businesses or other entities to receivemarketing or promotional materials. During selection of such input dataor lists (the provision of a “list of lists”), analytic recommendationprocessor 350 may provide data intelligence input (step 410), forexample, based on previous response rates of certain data sources.Continuing with the example, analytic recommendation processor 350 mayrecommend lists D and E, instead of F and G, as having provided higherresponse rates in past client marketing campaigns.

In addition, data selection and planning of step 405, with provided dataintelligence of step 410, will typically include attribute and featureselection. For example, various clients may determine that income, homeownership, and vehicle ownership are attributes which may be analyzed ormodeled to provide meaningful selection for promotions, such as formarketing materials for computer sales. In addition, analyticrecommendation processor 350 may suggest additional availableattributes, based on past response rates or other modeling, such asresponse rates through different communication channels (e.g., directmail, email, radio, television, etc.). Various orderings of processes orapplications may also be selected; in accordance with the invention,each such ordering, through the parameterized metadata of the invention,results in a different application being processed in the system 300.Such ordering, and how each process will be performed, is part of theservice orchestration provided by control systems 260. Other parameterswill also be selected or input, such as confidence levels, variousnumbers or amounts (e.g., the type and size of the resulting marketingcampaign, such as a catalogue mailing to 100,000 entities), selected orpreferred formats, and so on. The various selections are incorporatedinto the parameterized and linked metadata, allowing for customizationand individualization of the data processing, without coding changes.

It should be noted that a particularly novel feature of the presentinvention is that such data selection and planning (of steps 405 and410), is not a static and time-constrained process, but is dynamic andvariable, with changes and modifications allowed throughout the entiredata processing of the invention. For example, initial parameters may bechanged, in real-time and on the fly, with very limited reprocessingrequired. As indicated above, for data reduction, because interimresults are stored as retained relationships, changes to data reductionparameters may be made subsequently and quite rapidly, includingrepopulating or rebuilding the data mart 330 and analytics andsegmentation processing data structure 340. Indeed, using empiricalresults from data simulations for changes in the data reduction process,rather than taking approximately 3 hours to reprocess, in accordancewith the present invention, such reprocessing occurred in less than oneminute, resulting in over a 180-fold improvement. Such improvement ofmore than two orders of magnitude allows the various systems of thepresent invention to provide real-time analysis, unavailable in anyprior art systems.

Following initial data selection and planning, the data assembly processmay begin, step 415, and is typically performed by the data assemblyprocessor 315. As illustrated in FIG. 5, such data assembly involves awide variety of processes, illustrated as exemplary processes 415Athrough 415J, which may be individually selected or selected as needed,and which may occur in a wide variety of orders, depending upon thestatus of the data, its match capability, etc. Depending upon theselected embodiment, more or fewer of these processes 415A through 415Jmay be included, and may be performed as known or may become known inthe art. The data assembly process of step 415 may also be interactive,depending upon the selected embodiment, with user, client or operatorinput provided (e.g., through computers 175 or user/control node 135),and through analytic recommendation processor 350.

Data is input, step 415A, such as received electronically or throughinput media, such as magnetic tape or disk. The data layout isdetermined, step 415B, such that the various systems (100, 200 or 300)may properly process the various input data fields. Depending upon thedata, the data may be immediately matched and linkage (PINs) matched orassigned, as discussed above, step 415E; alternatively, this matchingand linkage assignment may occur in other orders, such as after dataconversion and correction. Also depending upon the data, data changedetection and update (step 415J) may also occur immediately or in otherorders, such as after data conversion and correction. Such changedetection, for example, provides for required processing of only changedrecords, with unchanged records not requiring additional processing.Data conversion and correction (often referred to as data “hygiene”) isperformed, step 415C, such as conversion to known or standard formats,and corrected as needed, such as by updating or correcting addresses.Data coding is performed, step 415F, such as by adding codes to improvemail deliverability and provide for postal discounts, along with theprovision of any data enhancements, step 415G, such as attachingadditional selected attributes or dimensions, such as modeling scores,vehicle ownership, etc. Data audits may also be performed, step 415D, asselected by the client or other user. For example, a particular clientor user may have a private repository 285, typically having a knownformat, which may then be audited to provide data corrections, updates,remove duplications, convert formats, and so on.

Typically, when multiple data sources have been input, such as multiplerented name lists, many of the same entities appear on more than onelist. Using PIN linkage selection, step 415H, only one such entity isselected, such that duplicate names or entities are not stored in datawarehouse 420 (providing the equivalent functionality to the prior artprocess of list merging and de-duplication, typically referred to as“merge/purge”). In addition, depending upon the selected embodiment, andas provided by parameters within the metadata, additional variables maybe created, such as by combining attributes to create new variables,dimensions or attributes, typically as part of the data enhancementprocess, step 415G.

These various processes may also be applied to the other types of datadiscussed above, such as for speech and signal processing. For example,data input sources may be selected in data selection and planning step405, such as speaker language selection, for speech recognition andsynthesis, from appropriate data repositories 412 and data sources 414,such as repositories storing recorded and digitized speech samples forthe selected language. The data assembly process 415 will also includedata input 415A, data layout 415B, data coding 415F (such as coding ofparticular phonemes with corresponding prosodies and co-articulations),and data enhancement 415G, such as for providing modified samples forconcatenation of phonemes to create appropriate co-articulations.

The data resulting from the data assembly process 415 is then stored indata warehouse 320, as step 420. For an exemplary campaign managementembodiment, the stored data will typically include entity name andaddress information, persistent identifiers, entity history (as coded ormaintained by retained relationships), and any other selected attributesand dimensions, and may be provided by any of the databases previouslydiscussed, e.g., by the various databases contained in partitioneddatabase storage 245. For an exemplary speech recognition or speechsynthesis embodiment, the stored data will typically be a plurality ofdigital signals corresponding to selected phonemes of a plurality ofdialects of one or more selected languages, with linkage to differentspeakers of different genders and ages, corresponding prosodies andco-articulations, for example.

Also in sharp contrast to the prior art, the data assembly process 415of the present invention is not a batch process which may take hours orweeks to complete, with all parameters required to be specified prior tocommencement of the assembly. Rather, such processes may be performed ona continuous basis, and may be performed in parallel and asynchronously;as data is input, it is processed and assembled, and moved into the datawarehouse 320. Multiple data streams may be processed in parallel, andat any time, without regard to when other data streams are beingprocessed. As a further consequence, and as discussed in greater detailbelow, other decisions may also be made later in the campaign managementprocess, in real-time and based on actual data segmentation andanalysis, providing a much more valuable process compared to the priorart. For example, the present invention allows the user to makedecisions based on “what if” scenarios, using actual data analysisprovided by the present invention. Effectively, use of the variousembodiments of the present invention allow what was formerly “batchprocessing”, occurring over several weeks, days or hours, to move intoreal-time processing.

Another novel feature of the present invention is that creation of acandidate data set has been completed with the creation of the datawarehouse 320. Using retained relationships, such as through the linkageprocess, for example, no rematching is required due to subsequentchanges in confidence levels. Through the linkage process, additionalinformation may be automatically available (albeit having a differentconfidence level), resulting in “inherited” relationships, without anyrequired reprocessing, as the complete relationship history has beenretained through the PIN assignment and matching process.

Following creation of the data warehouse 320 in step 420, the datareduction process may begin, step 425, and is typically performed bydata reduction processor 325. As illustrated in FIG. 5, such datareduction involves a wide variety of processes, illustrated as exemplaryprocesses 425A through 425D, which may be individually selected orselected as needed, and which may occur in a wide variety of orders,depending upon the status of the data, and so on. Depending upon theselected embodiment, more or fewer of these processes 425A through 425Cmay be included, and may be performed as known or may become known inthe art, but using the retained relationships feature of the presentinvention, step 425D.

The data netdown process or service 425A of the data reduction processtypically creates matches, superkeys and other PINs or relationships,and may eliminate various entities. For example, members of the samehousehold may be combined using a superkey or PIN, resulting in areduction of several entities into one entity. Similarly, depending uponselected confidence levels, entities having separate PINs may becombined, such as the individual “E. Tagliere” combined with theindividual “Eric Tagliere”, also reducing the number of entities. Thevarious entities eliminated in the data netdown process are stored,through their corresponding PINs, as a first data set, in one or more ofthe databases or memories, step 425D, such as stored in data warehouse420. The data suppression process or service 425B is utilized toeliminate entities based on any selected attribute, such as eliminatingentities based on postal codes, states, do not call lists, legal status(e.g., as a minor or as incarcerated), or previous inclusion in asolicitation, for example. Similarly, the various entities eliminated inthe data suppression process are stored, through their correspondingPINs, as a second data set, step 425D, also in one or more of thedatabases or memories, such as data warehouse 420. The dataprioritization (or hierarchy) process or service 425C is utilized toeliminate entities based on any selected priority; for example, anentity appearing on multiple lists may be selected as being from a lessexpensive list. Similarly, the various entities eliminated in the dataprioritization process are stored, through their corresponding PINs, asa third data set, step 425D, also in one or more of the databases ormemories, such as data warehouse 420. Similar data reduction processesare available in other embodiments, such as data netdown of speechsamples (phonemes) based on speaker age, gender, and language dialect,for example.

Also as illustrated in FIG. 5, the data reduction process of step 425may also be interactive, depending upon the selected embodiment, withuser, client or operator input (e.g., through computers 175 oruser/control node 135), and through analytic recommendation processor350. For example, analytic recommendation processor 350 may provide datareduction selections based on previous results, such as previousmarketing campaigns, or from various data models. Similarly, based onsubsequent results and segmentations from previous iterations of thisprocess, a client or operator may modify these data reduction processes,such as by adjusting metadata parameters.

As indicated above, the retained relationships, as one or more datasets, is one of the novel features of the present invention.Specifically, the retained relationships allow for rapid recomputations,based on any new or modified parameters. As indicated above, incomparison with the prior art, the retained relationships of the presentinvention have resulted in simulations having an improvement inprocessing speed of over two orders of magnitude.

The retained relationships also allow for parallel and asynchronousprocessing. For example, any of the various data reduction processes 425may be performed in parallel and at any time, without regard to whenother data reduction processes may occur, with interim results (as PINs)being stored in any of the various illustrated data storage devices.Following such parallel and asynchronous processing, final results maybe determined, such as through various set operations, such as formingan intersection of the interim results from data netdown, datasuppression, and data prioritization.

The results from the data reduction process are then stored in one ormore data marts, step 430, such as a data mart 330. For example,variations from each such reduction process may be stored in a separatedata mart 330, such as for multiple campaigns. The stored information,in selected embodiments, consists of PINs and selected attributes forsubsequent use in segmentation and analysis, because selection of a PINprovides an immediate index or key into all other information associatedwith the PIN, again, without reprocessing. As indicated above, a datamart 330 is typically implemented utilizing a star or snowflake schema.Following creation of the data mart 330, a data aggregation process iscommenced, step 435, using data aggregation processor 335, to provideone or more analytics and segmentation processing data structure 340,such as one or more multidimensional data cubes, illustrated as step440.

This data aggregation process of step 435 converts the relationalstructure of the data mart 330 to an aggregated, OLAP structure, usingany selected attributes or dimensions. The aggregation process is alsotypically interactive, with user/client input and intelligence providedby analytic recommendation processor 350, for selection of attributes ordimensions for aggregation. In addition, the aggregation process mayalso be iterative, with a given iteration based upon results fromprevious aggregations, and resulting segmentations, analysis, andreports. For example, selected dimensions for aggregations, as ranges ofattributes or other attribute values, may initially include gender, age,household income, home ownership, postal code, profession, vehicleownership by brand, and previous purchasing behavior by selectedretailer. Continuing with the example, initial dimensions for the incomeattribute may be (in thousands) 0-25, 26-50, 51-75, and so on;dimensions for age may be (in years) 0-10, 11-15, 16-20, and so on; anddimensions for gender may simply be binary, such as 0 for male and 1 forfemale. Aggregation is then performed, aggregating or combining all PINswhose corresponding attributes meet each dimension, for eachcorresponding intersection of the dimensions, such as each PINcorresponding to an individual of the selected gender in the selectedage range and the selected income range. Based on input from analyticrecommendation processor 350 or previous iterations, subsequentaggregating dimensions may include number of children and eliminatevehicle ownership as a dimension. Analytics may also be utilized in theaggregation process, such as based on resulting scores from applicationof one or more data models. Again, given the novel system structure ofthe present invention, such interactivity and attribute selectionchanges may be performed without reprocessing the data itself, and isavailable through the retained relationships and linkage of theinvention.

Following its creation, the analytics and segmentation processing datastructure 340 may be queried, step 445, with the return of queryresults, step 450, in both an interactive and iterative process. Forexample, with user or operator interaction, step 455, and withintelligence input from analytic recommendation processor 350, theanalytics and segmentation processing data structure 340 may berepeatedly queried, analyzing different segments of the OLAP structure,such as various intersections, slices or segments, to determine anoptimal selection of entities, what information should be provided tothem, and over what communication channel, for example. At each stage, areport may be generated, step 460, for use in additional analysis, or asa final report of such interactive selections.

Use of such an OLAP structure as analytics and segmentation processingdata structure 340 is completely new to this voluminous and complex dataprocessing, such as in campaign management. Previously, such campaignmanagement has also been confined to mainframe computing involving setmanipulations confined to large relational database structures. Inaccordance with the exemplary embodiments, OLAP structures have now beenextended into data set theory and set manipulations, rather than merelyarithmetic calculations such as sums and averages. In addition, asdiscussed in greater detail below with reference to FIG. 9, the presentinvention also provides for modified-OLAP functionality, to combine OLAPaggregation and segmentation capabilities with both set algebraicmanipulations and operations, and with logical operations which provideprocessing to simulate set operations.

In addition, as another novel feature of the present invention, theaggregation of the present invention into the analytics and segmentationprocessing data structure 340 having an OLAP structure provides forreal-time and modifiable segmentation and analysis, using any of theselected dimensions or attributes utilized in the aggregation, allowingany corresponding roll-up or drill-down operations in dimensions havingthose attributes, and again, having an interactive and iterativecapability, using query determination (step 445) and query results (step450). For example, a given analysis may roll-up geographic segments fromhousehold to postal code level; conversely, an analysis may drill-downin a geographic segment from state level to postal code level, or from ahousehold level to an individual level.

For example, in addition to the set intersection operations involved inthe aggregation of entities based upon the selected attributes for theOLAP dimensions, exemplary embodiments of the present invention alsoprovides set summation operations, to provide for roll-up operations forthe corresponding dimensions. As discussed in greater detail below, theexemplary embodiments provide entirely new OLAP functions, namely, setunion and set difference operations. As a result, the exemplaryembodiments provide for real-time “what if” analyses, heretoforeunavailable in the art of marketing campaign management.

Yet another novel feature of the present invention is the movement ofthe decision-making process to the end of the campaign determinationprocess. In the prior art, all such decisions had to be made at thecommencement of the data processing for the marketing campaign, prior tothe performance of batch processing, including selection of segmentationparameters or attributes. In accordance with the present invention,however, segmentation, analysis and modeling may all be performed inreal-time, iteratively, interactively, and using actual data.

Once determinations have been made from the previous analytical steps,the process continues with fulfillment processing, step 465, typicallyperformed by the fulfillment processor 345. For example, variousfulfillment processes may be performed, such as generation of a magnetictape, disk or electronic file having selected names and physicaladdresses, email addresses, with corresponding selected content andcommunication channel (e.g., who gets what offer or other communicationand on what channel), selected formatting, postal presorting, inkjetservices (e.g., mailing labels), and actual shipping or other execution,such as provision of emailing services.

The process may also include follow up procedures, such as responsedetermination and modeling, step 470, which may be utilized insubsequent iterations, as discussed above, such as for feedback into thequery process, as illustrated. Following steps 465 or 470, the methodmay end, return step 475.

Similar functions may be performed in the exemplary speech recognitionand synthesis embodiment. For example, the aggregation process 435 mayinclude dimensions based on dialect, gender, age, selected prosody, andso on, with subsequent analysis based on likelihood of fit of receivedspeech to selected phoneme strings stored in the analytics andsegmentation processing data structure 340.

The various systems (100, 200 and 300) of the present invention, whileillustrated in greater detail with respect to marketing campaignmanagement, may also be extended into other areas, such as customermanagement. For example, customer data may be merged with prospect datainto a data warehouse 320, followed by the real-time segmentation,analysis and modeling of the present invention, for improved predictioncapabilities. In addition, using the metadata linkage structure of thepresent invention, discussed below with reference to FIGS. 7 and 8,additional layers of functionality may be incorporated in a modularfashion, without requiring changes to the existing system, such as theaddition of a wide variety of potential data modeling systems.

In addition, through the use of parameterized metadata, solutions may becustomized, without requiring any hard recoding. For example, asmentioned above, confidence levels for matching requirements, and thevarious forms of data reduction of step 425, are provided usingcustomizable parameters. In addition, attribute and dimension selection,for segmentation and analysis, are also provided based on customizableparameters. This parameterized metadata, and a new type of linkageprovided at a metadata level, are discussed in greater detail below.

FIG. 7 is a block diagram illustrating an exemplary metadata embodimentfor an exemplary system embodiment, utilized in a marketing campaignmanagement application, in accordance with the teachings of the presentinvention. As indicated above, the metadata of the present inventiondefines both data structures and data processes, allowing both to beaccessed together in a relational form, with selectable parameters. Theexemplary metadata is provided in a modular form, and allows versioningof the processes and services provided by the systems (100, 200 or 300),through parameter selection, allowing the creation of different processflows, the passing of information between processes, as parameters orreturn conditions, the linkage between different processes, and theprovision of conditional services. For example, each process is given aname and stored in metadata, such that “services” being provided by theexemplary systems are metadata processes having parameters, definedinputs, and defined outputs, which may also link into other metadatamodules. In addition, and as mentioned above, the metadata of thesystems (100, 200 or 300) is an open model, allowing the introduction ofadditional services in a modular, “plug and play” manner. In exemplaryembodiments, the metadata defines data structures in third normal form.

FIG. 7 further illustrates modular metadata, with linkage createdbetween such modules (“metadata linkage”), such as through a shared orcommon parameter, a common process or process definition, a commonprocess element or process flow, or a shared relationship or sharedattribute. As illustrated, the various modules 600, 605, 610, 615, 620,and 625, with their corresponding metadata linkages, are generallyapplicable to any form of data processing (e.g., are generic for theexemplary system architectures), while the remaining modules are morespecific and are illustrated as exemplary of a marketing campaignmanagement or service bureau application. Similar metadata modules maybe created for other types of data processing, such as for the variousspeech and signal processing applications discussed above. In should benoted that each of the various illustrated modules may themselves becomprised of additional modules and, conversely, may also be combined toform larger modules, all of which are within the scope of the presentinvention.

Referring to FIG. 7, a first metadata module 600 defines services(processes) and process flows which may be undertaken by the systems(100, 200 or 300), such as the various processes discussed above, e.g.,data assembly. As part of this module 600, processes may be selected orspecified, along with their orders and dependencies. Such servicedefinitions are invoked through metadata link 602, linkingservice/process definitions to service/process instances, by secondmetadata module 605, to create a service instance (or instantiation),and to provide a record of what processes are occurring and when. Inturn, each of these various services are comprised of correspondingactivities (and lists of activities) defined in third metadata module610, such as the repeated execution of matching processing, which arelinked (metadata link 607), to correlate such services with theircorresponding activities. For example, services being performed for agiven client may involve multiple activities, such as a winter campaign(defined in 630) involving 10 different activities (610), for example,all linked to the corresponding services (600).

In addition, each of these various services utilize hardware resourcesdefined in fourth metadata module 615, and are linked through metadatalink 609, which provides resource assignment to the various services.Through these linkages, usage of various processes and hardware aretracked, allowing analysis of resource usage, which may also be utilizedas an indication of an associated value of a process or activity (forexample, activities which are invoked frequently may have more valuethan those invoked infrequently), and for planning for future growth. Inturn, both third metadata module 610 and fourth metadata module 615 arelinked (metadata links 612 and 617) to accounting services defined infifth metadata module 620, to respectively correlate activities andresources to their corresponding costs and corresponding client billing,and further to provide cost modeling and capacity planning (metadatalink 617). These general metadata modules are linked to the parties onwhose behalf services and activities are being performed, as defined insixth metadata module 625, through metadata link 622, providing costallocations to corresponding parties (clients).

Similarly, for the exemplary marketing campaign management embodiment,parties such as clients or other users specified in sixth metadatamodule 625 are linked (metadata link 627) to their specified marketingcampaigns defined and selected in seventh metadata module 630, such asthrough selection of the various parameters and processes discussedabove. The sixth metadata module 625 is also linked (metadata link 622)to the fifth metadata module 620, to associate parties and correspondingaccounting. The seventh metadata module 630 will indicate the type ofmarketing campaign, how frequently it will be performed, the campaigngoals, and corresponding budgets. The seventh metadata module 630defining a marketing campaign is also linked (metadata link 629) to thecorresponding detailed activities comprising a selected campaign anddefined in third metadata module 610, and linked (metadata link 631(list assignment to campaigns)) to eighth metadata module 635, whichdefines the “list of lists” (i.e., list of data inputs) which can beselected and used as data input into a given campaign, matchingcampaigns to lists, for payment of list brokers, and for list selection.The eighth metadata module 635 is also linked (metadata link 637 (listusage by parties)) to the corresponding activities defined in thirdmetadata module 610 which will be utilized in processing the selectedlists (as different lists will generally have different associatedactivities), such as use of a first list for enhancement activity, anduse of a second list for suppression activity; further linked (metadatalink 639 (list usage by parties)) to the parties of sixth metadatamodule 625 utilizing the corresponding selected lists (as not everyparty will have access to all lists, such as private lists belonging toanother party); and also linked (metadata link 643 (detailed listlayouts and audits)) to the corresponding lists, data layouts and auditsdefined in ninth metadata module 640. For example, the ninth metadatamodule 640 defines, for each list or other data input which may beselected, the detailed data layouts and fields which will be processed.Similarly, parties such as clients or other users specified in sixthmetadata module 625 are linked (metadata link 647 (party-specificinteraction)) to their selected interactions specified in tenth metadatamodule 645, such as client-specific list priorities and suppressionrules, and linked (metadata link 652 (party-specific enhancements)) totheir selected data enhancements specified in eleventh metadata module650, such as which data aggregations will be performed and whichattributes included in formation of the client's data warehouse 320.

FIG. 8 is a diagram illustrating in greater detail exemplary metadatamodule embodiments and metadata linkages for an exemplary systemembodiment in accordance with the teachings of the present invention,and illustrates various entities, relations, and metadata linkagesdescribed above. As illustrated, Service entity or module 655 definesvarious services, service versions, cost models, and valid dates. Module655 then has a one or zero to many relation (link 656) to Called Serviceentity or module 665, defining services and versions, and has a one orzero to many relation (link 657, linking services to parties) to PartyService Invocation 670, which defines or references parties, theirservices and service versions, the reason the service was invoked, andthe scheduled and actual start times of the service. Party entity ormodule 660, defining parties, party type and cost center, also has a oneor zero to many relation (link 661) to Party Service Invocation module670, where the relation is not a dependency (indicated by the dashedline). Party Service Invocation module 670 is also linked by a one orzero to many relation (link 671) to Process entity or module 675, whichdefines processes, occurring at a given IP address, its timestamp, withadditional links or relations to parties, services, service versions,and service invocations. Server entity or module 680 defining the serverIP addresses, types and descriptions, which perform the variousprocesses, in turn, also has a one or zero to many relation (link 681)to Process entity or module 675.

The metadata linkage and metadata parameterization of the presentinvention, as illustrated above for a selected application for marketingcampaigns, provides a very significant advantage to the systems (100,200 or 300). Most significantly, any such campaign may be customized byselection of corresponding parameters, which are then automaticallylinked throughout the system with referential integrity, without anycoding changes, and without any user or operator intervention.Similarly, another service or functionality, such as modeling, may beincorporated into the systems (100, 200 or 300) by providing anadditional metadata module, and corresponding linkage into at least oneother module (which, in turn, provides pre-existing metadata linkagewith the remaining modules). As a result, the metadata parameterizationand the metadata linkage of the present invention allow fullcustomization of the marketing campaign process while retaining fullautomation, and further allows modular expansion of the system to addnew functionality.

Significantly, the metadata linkages of the present invention allow forelimination of custom program code writing for performance of dataprocessing. The metadata linkages and parameterized metadata insteadprovide for implementation of generic rules to define a data process,with processes (such as a marketing campaign or a speech recognitionprocess) modeled in a relational form, with the referential integritybetween services/processes and selectable parameters utilized tocustomize the selected data process. As a consequence, selection ofparameters and services results in automatic and customized processcreation.

Those of skill in the art will recognize that the metadata of theexemplary embodiments of the present invention is in sharp contrast withprior art metadata, which has never been utilized to define not onlydata structures, but also to define processes and dynamic and staticparameters. This new form of metadata allows a new level ofgeneralization, with customization available through both processselection and parameter selection, without the prior art need forrecoding or new code generation. Such metadata is also integral to theexemplary systems and processes, rather than merely providing tangentialinformation (e.g., media “metadata” such as song title and performer ofU.S. Patent Application Publication No. 2005/0015389 assigned toMicrosoft Corp.). Similarly, the exemplary metadata of the presentinvention with explicit parameter selection for complete processcustomization is wholly dissimilar from other attempts at customizingprior art metadata (e.g., attempted customization of metadata throughthe use of wholly static metadata, with differences temporarily storedas “deltas” from the static metadata, illustrated in U.S. PatentApplication Publication No. 2004/0181534 also assigned to MicrosoftCorp.).

OLAP data structures, as indicated above, are adapted to provide rapidsegmentation capabilities through pre-aggregation of data based onselected attributes or dimensions. Selection of multiple attributes ordimensions, moreover, results in the determination of an intersection ofthese various dimensions, such as a first set of PINs (as set “X”)corresponding to individuals, for example, having annual incomes between$100,000 and $150,000, who drive Volvo automobiles, have 2 pets, andlisten to National Public Radio's “All Things Considered” during theirworkday commute. Similarly, segmentation using different attributes ordimensions may result in a second set of PINs (as set “Y”) correspondingto individuals, for example, who are married with 2 children, own theirown home, watch cable and satellite television news programs, and preferemail communications. In addition, segmentation using differentattributes or dimensions may result in a third set of PINs (as set “Z”)corresponding to individuals, for example, who are married, own theirown home, own a vacation home, travel to the Caribbean every winter, andprefer postal communications.

What OLAP data structures are not adapted to provide, however, is thegroup of individuals who are in set X, the group of individuals who arein set Y and not in set X, and the group of individuals who are in set Zand not in either set X or set Y, namely, mutually exclusive setoperations, such as set difference operations and set union operations(without duplicate entries). For example, a user may want to conduct amailing to individuals in sets X, Y and Z, but does not want to sendduplicate communications to individuals who are in 2 or more sets. Usingprior art OLAP data structures, such a determination grows exponentiallywith the number of sets involved. For example, using the segmented dataavailable in an OLAP cube, to find the union of the 3 sets, 7 setdeterminations are required, as set X, set Y, set Z, set X intersectionY, set X intersection Z, set Y intersection Z, and set X intersection Yintersection Z, followed by set operations adding the individual sets,subtracting the paired intersections, and adding back in the tripleintersection, i.e., X∪Y∪Z=(X+Y+Z)−(X∩Y)−(X∩Z)−(Y∩Z)+(X∩Y∩Z).

As a consequence, in accordance with the present invention, amodified-OLAP functionality is provided, in which OLAP aggregation andsegmentation capabilities are combined with both algebraic manipulationsand set operations to create union and difference operations. Using themodified-OLAP functionality of the present invention significantlyreduces the number of required calculations, such that determinationsonly grows arithmetically with the number of sets involved, and notexponentially.

FIG. 9 is a flow diagram illustrating an exemplary method embodiment forset determination for modified-OLAP functionality in accordance with theteachings of the present invention. The method is typically embodied asa suitably adapted or programmed processor of the plurality ofprocessing nodes (205, 210, 215, 220) or within other processors andprocessing nodes referred to as a data assembly processor 315, datareduction processor 325, aggregation processor 335, or fulfillmentprocessor 345. The method begins, start step 700, with determination ofan initial set, step 705, such as a set “X”. Each of the various setsdetermined by the method, including interim sets, are also stored in amemory (e.g., 120) or other data storage devices (e.g., 145, 240, 245,250, 255), typically as a set of PINs. A next set is determined, step710, such as set “Y”. A first union set is created as a de-duplicatedcombination of the initial set and the next set (i.e., X∪Y), step 715,where duplication is eliminated by selection of only one PIN where thePIN is represented in more than one set. A first difference set iscreated as the first union set minus the first set (i.e., (X∪Y)−X), step720, providing a difference set comprising those elements of set Y whichare not also in set X.

Next, when additional sets are to be determined, step 725, a next set isdetermined in step 730, such as set “Z”. A next union set is created asa de-duplicated combination of the previous union set and the next set(i.e., (X∪Y)∪Z), step 735. A next difference set is created as the nextunion set minus the previous union set (i.e., [(X∪Y∪Z)−(X∪Y)], step 740,providing a next difference set comprising those elements of set Z whichare not also in sets X or Y. When there are additional sets to bedetermined, the method continues, step 725, returning to steps 730, 735and 740, to continue to determine difference sets from the next unionset minus the previous union set, creating a set of PINs which are notin any of the previously determined sets. When there are no more sets tobe determined in step 725, the method provides or outputs the initialset and all of the one or more difference sets which have beencalculated, step 745, and the method may end, return step 750. As aconsequence, in the exemplary systems, a processing node is adapted todetermine a plurality of mutually exclusive sets from a plurality ofsequential union sets by forming a difference set by subtracting aprevious union set from a next union set.

FIG. 10 is a block diagram of fourth system 800 embodiment having aplurality of ASICs in accordance with the teachings of the presentinvention, and provides a useful summary. As illustrated, the system 800comprises a data interface 805 (such as an I/O 185 or 105), an assembler810, a reducer 815, an aggregator 820, a set operator 825, an outputgenerator 830, all embodied as a plurality of ASICs, and a memory 835,which may be any form of memory discussed above (such as discussed withreference to memory 120). The data interface 805 is utilized for datainput and output, and may be any kind or type of interface or I/O, suchas a network interface. The assembler 810 performs the functionalitydiscussed above with respect to the data assembly processor 315, andsteps 415A-415J of the method embodiment. The reducer 815 performs thefunctionality discussed above with respect to the data reductionprocessor 325, and steps 425A-415D of the method embodiment. Theaggregator 820 performs the functionality discussed above with respectto the aggregation processor 335 and step 435 of the method embodiment.The set operator 825 performs the functionality discussed above withrespect to the set operations provided in the plurality of processingnodes (205, 210, 215, 220), such as the set union and differenceoperations discussed above with reference to FIG. 9, and the setintersection operations using the plurality of interim results from thedata reducer, referred to above, for example. The output generator 830performs the functionality discussed above with respect to thefulfillment processor 345 and step 465 of the method embodiment.Additional memory 840 may be provided, typically not as part of the SOC800, for data storage for the data of the various data repositories anddata structures of the present invention. In selected embodiments, aplurality of the systems 800 may be utilized in parallel, to providehigh performance processing.

It will be apparent to those of skill in the art that while the variousembodiments of the invention have been described primarily with respectto applications involved in marketing campaign management, the varioussystems and methods of the invention may be extended to numerous otherfields, including other technical fields such as speech and signalprocessing.

For example, the present invention can be applied to speech, optical andsignal processing, where vast quantities of data are collected and mustbe analyzed. For accurate speech recognition, vast data stores may begenerated, for thousands of electronic signals. These electronic signalmust be stored and associated with corresponding phonemes, for thousandsof words, each from thousands of individuals, in many differentlanguages. Again, vast quantities of data must be processed, and mustresult in a selection of words and corresponding pronunciations thatmeet certain criteria, such as having a likelihood of fit to selectedphoneme patterns from a plurality of different speakers of a pluralityof different languages, with high discrimination and noise immunity.

The various embodiments of the present invention provide for such signalprocessing and analysis, in real-time. Input signals may be sampled,categorized, matched to phonemes, and assembled into a data warehouse.Similarly, for optical character recognition, input signals are sampled,categorized, matched to expected characters, and assembled into a datawarehouse. For subsequent analysis, using selection criteria andattributes, such as selected phonemes of particular dialects, the datamay be reduced and stored in a data mart. This data may then beaggregated for a multidimensional data structure, which then supportsreal-time query processing for real-time data analysis, for use inspeech recognition devices.

From the foregoing, it will be observed that numerous variations andmodifications may be effected without departing from the spirit andscope of the novel concept of the invention. It is to be understood thatno limitation with respect to the specific methods and apparatusillustrated herein is intended or should be inferred. It is, of course,intended to cover by the appended claims all such modifications as fallwithin the scope of the claims.

1. A tangible medium storing computer readable program instructions forexecution by a computer system for marketing campaign management andanalysis, the tangible medium storing computer readable programinstructions comprising: a first metadata construct of a plurality ofmetadata constructs, the first metadata construct defining a pluralityof data processes for marketing campaign management; a second metadataconstruct of the plurality of metadata constructs, the second metadataconstruct having a corresponding metadata linkage to the first metadataconstruct, of a plurality of metadata linkages, the second metadataconstruct defining a first plurality of relational data structures for afirst plurality of data repositories of the computer system, the secondmetadata construct further defining a multidimensional data structurefor a second data repository for marketing campaign analysis; and athird metadata construct of the plurality of metadata constructs, thethird metadata construct having a corresponding metadata linkage to thefirst metadata construct, of the plurality of metadata linkages, thethird metadata construct defining a plurality of input data sources anda plurality of selectable attributes.
 2. The tangible medium of claim 1,further comprising: a fourth metadata construct of the plurality ofmetadata constructs, the fourth metadata construct having acorresponding metadata linkage to the first metadata construct, of theplurality of metadata linkages, the fourth metadata construct adapted toprovide a plurality of modifiable parameters to the first metadataconstruct, the plurality of modifiable parameters providing forselection of the plurality of input data sources and selection of afirst subset of the plurality of data processes for execution by thecomputer system.
 3. The tangible medium of claim 2, wherein theplurality of selectable input data sources comprise a plurality ofdatabases and marketing lists containing a plurality of attributes, theplurality of attributes comprising at least two attribute types of thefollowing group of attribute types: a name attribute, an addressattribute, a demographic attribute, a behavioral attribute, or anattitudinal attribute.
 4. The tangible medium of claim 2, furthercomprising: a fifth metadata construct of the plurality of metadataconstructs, the fifth metadata construct adapted to execute the firstsubset of the plurality of data processes in parallel andasynchronously, using the plurality of modifiable parameters and acorresponding plurality of metadata linkages to other metadataconstructs.
 5. The tangible medium of claim 2, wherein the first subsetof the plurality of data processes comprises at least two data assemblyprocesses of the following group of data assembly processes: a datainput process, a data layout process, a data conversion process, a datacorrection process, a data audit process, a data matching process, alinkage assignment process, a linkage selection process, a data codingprocess, a data enhancement process, a data change detection process, ora data update process.
 6. The tangible medium of claim 5, wherein asecond subset of the plurality of data processes comprises at least onedata reduction process of the following group of data reductionprocesses: a data netdown process, a data suppression process, or a dataprioritization process.
 7. The tangible medium of claim 1, wherein athird subset of the plurality of data processes are adapted to use themultidimensional data structure to process a received query and toprovide a query response.
 8. The tangible medium of claim 1, furthercomprising: a sixth metadata construct of plurality of metadataconstructs, the sixth metadata construct adapted to track each dataprocess invocation of a plurality of data process invocations.
 9. Thetangible medium of claim 8, further comprising: a seventh metadataconstruct of the plurality of metadata constructs, the seventh metadataconstruct adapted to associate a cost factor with each data processinvocation of a plurality of data process invocations.
 10. The tangiblemedium of claim 8, further comprising: an eighth metadata construct ofthe plurality of metadata constructs, the eighth metadata constructadapted to associate a resource usage factor with each data processinvocation of a plurality of data process invocations.
 11. The tangiblemedium of claim 1, further comprising: a ninth metadata construct of theplurality of metadata constructs, the ninth metadata construct defininga plurality of marketing campaigns, each marketing campaign of theplurality of marketing campaigns associated with a correspondingselection of data processes of the plurality of data processes, and eachmarketing campaign of the plurality of marketing campaigns associatedwith a corresponding party of a plurality of parties.
 12. The tangiblemedium of claim 1, further comprising: a tenth metadata construct of theplurality of metadata constructs, the tenth metadata construct adaptedto use the plurality of selectable attributes for a dimensioning andaggregating data for storage in the multidimensional data structure. 13.The tangible medium of claim 1, wherein the plurality of metadatalinkages comprises at least one metadata linkage type of the followinggroup of metadata linkage types: a common or communicated parameter ofthe plurality of modifiable parameters, a common or communicatedattribute of a plurality of attributes, a common process definition, acommon process flow, or a communication between processes.
 14. Thetangible medium of claim 1, further comprising: an eleventh programconstruct adapted to determine a plurality of available variations of atleast one of the following: selectable data processes, selectable dataattributes, or selectable data segmentations.
 15. The tangible medium ofclaim 1, further comprising: a twelfth program construct adapted toprovide data modeling and data analysis using the plurality ofmodifiable parameters, the plurality of selectable input data sources,and the plurality of selectable attributes.
 16. The tangible medium ofclaim 1, further comprising: a thirteenth program construct adapted toprovide at least one selection recommendation for the plurality ofmodifiable parameters, the plurality of selectable input data sources,or the plurality of selectable attributes.
 17. A tangible medium storingcomputer readable software for programming a parallel and asynchronousdatabase architecture and data processing system for marketing campaignmanagement and analysis, the tangible medium storing computer readablesoftware comprising: a first program construct of a plurality of programconstructs, the first program construct comprising a plurality ofmodifiable parameters adapted to be modifiable during data processing,the first program construct further defining a plurality of selectableinput data sources and a plurality of selectable attributes; a secondprogram construct of the plurality of program constructs, the secondprogram construct having a corresponding metadata linkage to the firstprogram construct, the second program construct defining a plurality ofdata structures for a plurality of data repositories of the computersystem, the plurality of data structures comprising at least onerelational data structure and at least one multidimensional datastructure; a third program construct of the plurality of programconstructs, the second program construct having corresponding metadatalinkages to the first and second program constructs, the third programconstruct adapted to provide for execution of a plurality of dataprocesses using the plurality of modifiable parameters, the plurality ofinput data sources and the plurality of selectable attributes, the thirdprogram construct further adapted to store a plurality of results of theplurality of data processes in the plurality of data structures; and afourth program construct of the plurality of program constructs, thefourth program construct having corresponding metadata linkages to thefirst, second and third program constructs, the fourth program constructadapted to use the plurality of selectable attributes for a dimensioningand aggregating data for storage in the multidimensional data structure.18. The tangible medium of claim 17, wherein: a first subset of theplurality of data processes comprises at least two data assemblyprocesses of the following group of data assembly processes: a datainput process, a data layout process, a data conversion process, a datacorrection process, a data audit process, a data matching process, alinkage assignment process, a linkage selection process, a data codingprocess, a data enhancement process, a data change detection process, ora data update process; and a second subset of the plurality of dataprocesses comprises at least one data reduction process of the followinggroup of data reduction processes: a data netdown process, a datasuppression process, or a data prioritization process.
 19. The tangiblemedium of claim 18, wherein the data assembly processes and datareduction processes correspondingly provide for data assembly and datareduction on the basis of the plurality of selectable attributes. 20.The tangible medium of claim 17, wherein the plurality of data processesare adapted to execute in parallel and asynchronously using a pluralityof data processing nodes of the data processing system.
 21. The tangiblemedium of claim 20, wherein the third program construct is furtheradapted to provide for storage in a memory or in a database of acorresponding plurality of interim results from the asynchronous andparallel execution of the plurality of data processes.
 22. The tangiblemedium of claim 21, wherein the third program construct is furtheradapted to provide for the storage of the corresponding plurality ofinterim results as a corresponding set of unique identifiers.
 23. Thetangible medium of claim 17, further comprising: a fifth programconstruct of the plurality of program constructs, the fifth programconstruct having a corresponding metadata linkage to the third programconstruct, the fifth program construct adapted to track each dataprocess invocation of a plurality of data process invocations.
 24. Thetangible medium of claim 23, further comprising: a sixth programconstruct of the plurality of program constructs, the sixth programconstruct having a corresponding metadata linkage to the fifth programconstruct, the sixth program construct adapted to associate a costfactor and a usage factor with each data process invocation of aplurality of data process invocations.
 25. The tangible medium of claim17, further comprising: a seventh program construct of the plurality ofprogram constructs, the seventh program construct adapted to determine aplurality of sets resulting from the dimensioned and aggregated data inthe multidimensional data structure and to perform a plurality of setoperations on the plurality of sets, the plurality of set operationscomprising union operations, difference operations, and intersectionoperations.
 26. The tangible medium of claim 17, wherein the pluralityof selectable input data sources comprises a plurality of databases andmarketing lists containing a plurality of attributes, the plurality ofattributes comprising at least two attribute types of the followinggroup of attribute types: a name attribute, an address attribute, ademographic attribute, a behavioral attribute, or an attitudinalattribute.
 27. The tangible medium of claim 17, wherein the plurality ofmodifiable parameters further define a plurality of selectable dataenhancements.
 28. The tangible medium of claim 17, wherein the fourthprogram construct is further adapted to use the multidimensional datastructure to create a response to an input query.
 29. The tangiblemedium of claim 17, further comprising: an eighth program constructadapted to determine a plurality of available variations of at least oneof the following: selectable data processes, selectable data attributes,or selectable data segmentations.
 30. The tangible medium of claim 17,further comprising: a ninth program construct adapted to provide datamodeling and data analysis using the plurality of modifiable parameters,the plurality of selectable input data sources, and the plurality ofselectable attributes
 31. A tangible medium storing computer readableprogram instructions for execution by a computer system for marketingcampaign management and analysis, the tangible medium storing computerreadable program instructions comprising: a first program moduledefining a plurality of modifiable parameters adapted to be modifiableduring data processing and a plurality of selectable attributes; asecond program module defining a plurality of selectable input datasources, the plurality of selectable input data sources comprising atleast one consumer database and a plurality of consumer lists ofindividuals and households; a third program module defining a pluralityof data structures for a plurality of data repositories of the computersystem, the plurality of data structures comprising a first relationaldata structure, a second relational data structure storing a subset ofinformation from the first relational data structure, and at least onemultidimensional data structure; a fourth program module adapted toexecute a plurality of marketing campaign management data processesusing the plurality of modifiable parameters, the plurality ofselectable input data sources and the plurality of selectableattributes, the fourth program construct further adapted to perform afirst plurality of data assembly processes and store assembled data inthe first relational data structure, and to perform a second pluralityof data reduction processes on the assembled data and store reduced datain the second relational data structure; a fifth program module adaptedto use the plurality of selectable attributes for dimensioning andaggregation of the reduced data for storage in the multidimensional datastructure; a sixth program module to correlate invocation of theplurality of marketing campaign management data processes withaccounting functions and resource utilization; and a seventh programmodule to track selection of the plurality of consumer lists.
 32. Atangible medium storing a plurality of metadata modules for execution bya computer system for marketing campaign management and analysis, thetangible medium comprising: a first metadata module of a plurality ofmetadata modules, the first metadata module comprising a plurality ofmodifiable parameters adapted to be modifiable during data processing; asecond metadata module of the plurality of metadata modules, the secondmetadata module defining a plurality of selectable input data sourcescomprising at least one consumer database and a plurality of consumerlists; a third metadata module of the plurality of metadata modules, thethird metadata module defining a plurality of data structures for aplurality of data repositories of the data processing system, theplurality of data structures comprising a first relational datastructure storing a first subset of information, a second relationaldata structure storing a second subset of information from the firstrelational data structure, and at least one multidimensional datastructure storing a third subset of information from the secondrelational data structure which has been dimensioned and aggregated; afourth metadata module of the plurality of metadata modules, the fourthmetadata module having corresponding metadata linkages to the first,second and third metadata modules, the fourth metadata module adapted toexecute a plurality of data assembly processes using the plurality ofselectable input data sources and the plurality of modifiable parametersto form the first subset of information, the fourth metadata modulefurther adapted to execute a plurality of data reduction processes usingplurality of modifiable parameters to form the second subset ofinformation; a fifth metadata module of the plurality of metadatamodules, the fifth metadata module adapted to dimension and aggregatethe second subset of information to form the third subset ofinformation, using a plurality of selectable attributes; a sixthmetadata module of the plurality of metadata modules, the sixth metadatamodule adapted to perform a plurality of set operation on the thirdsubset of information using the plurality of selectable attributes; aseventh metadata module of the plurality of metadata modules, theseventh metadata module adapted to utilize the multidimensional datastructure to process a query and to generate a response to the query;and an eighth metadata module of the plurality of metadata modules, theeighth metadata module adapted to provide at least one selectionrecommendation for the plurality of modifiable parameters, the pluralityof selectable input data sources, or the plurality of selectableattributes.
 33. The tangible medium of claim 32, further comprising: aninth metadata module of the plurality of metadata modules, the ninthmetadata module adapted to track each invocation of a data process ofthe plurality of data assembly processes and the plurality of datareduction processes, and to associate each data process invocation withan accounting function and with a resource usage function.
 34. Thetangible medium of claim 32, wherein: the plurality of data assemblyprocesses comprises at least two data assembly processes of thefollowing group of data assembly processes: a data input process, a datalayout process, a data conversion process, a data correction process, adata audit process, a data matching process, a linkage assignmentprocess, a linkage selection process, a data coding process, a dataenhancement process, a data change detection process, or a data updateprocess; and the plurality of data reduction processes comprises atleast one data reduction process of the following group of datareduction processes: a data netdown process, a data suppression process,or a data prioritization process.
 35. The tangible medium of claim 32,wherein the plurality of selectable attributes comprises at least twoattribute types of the following group of attribute types: a nameattribute, an address attribute, a demographic attribute, a behavioralattribute, or an attitudinal attribute.
 36. A tangible medium storingcomputer readable software for programming a parallel and asynchronousdatabase architecture and data processing system for marketing campaignmanagement and analysis, the tangible medium storing computer readablesoftware comprising: a first program construct of a plurality of programconstructs, the first program construct comprising a plurality ofmodifiable parameters adapted to be modifiable during data processing; asecond program construct of a plurality of program constructs, thesecond program construct having a corresponding metadata linkage to thefirst program construct, the second program construct adapted to obtaina first modifiable parameter of the plurality of modifiable parametersfrom the first program construct, the second program construct furtheradapted to provide for execution of a first data process of a pluralityof data processes using the first modifiable parameter; a third programconstruct of the plurality of program constructs, the third programconstruct having a second metadata linkage to the first programconstruct, the third program construct adapted to obtain a secondmodifiable parameter of the plurality of modifiable parameters from thefirst program construct, the third program construct adapted to executea multithreaded data process of the plurality of data processes, inparallel and asynchronously using the second modifiable parameter,through a plurality of third metadata linkages to a subset of programconstructs of the plurality of program constructs, wherein a selectedprogram construct of the subset of program constructs is further adaptedto perform a set intersection operation on a corresponding plurality ofsets of unique identifiers resulting from the asynchronous and parallelexecution of the multithreaded process; and wherein a first subset ofmodifiable parameters of the plurality of modifiable parameters define aplurality of selectable input data sources, a second subset ofmodifiable parameters of the plurality of modifiable parameters define aplurality of selectable attributes, and wherein the plurality of dataprocesses comprise at least two of the following data processes: a dataassembly process, a data reduction process, a data aggregation process,or a data fulfillment process.